Jul 07 16:21:03.839019 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 16:21:03.839031 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 16:21:03.839038 ip-10-0-140-28 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 16:21:03.839257 ip-10-0-140-28 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 16:21:15.037626 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 16:21:15.037643 ip-10-0-140-28 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7a33ac0c99ea4f218941b80506177012 --
Jul 07 16:23:21.322057 ip-10-0-140-28 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.748478    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752241    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752253    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752257    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752261    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752264    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752267    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752271    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.768918 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752274    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752276    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752292    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752297    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752301    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752304    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752307    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752310    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752313    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752317    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752328    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752331    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752334    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752351    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752354    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752356    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752359    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752362    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752364    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.769738 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752367    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752369    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752372    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752374    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752377    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752379    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752382    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752384    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752387    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752389    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752391    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752394    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752396    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752399    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752403    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752405    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752409    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752412    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752415    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752418    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.770344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752420    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752423    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752426    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752428    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752431    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752433    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752436    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752445    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752448    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752451    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752453    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752456    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752458    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752461    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752463    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752465    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752468    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752470    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752473    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752475    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.770963 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752477    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752480    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752483    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752485    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752489    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752491    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752494    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752496    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752500    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752503    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752506    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752508    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752511    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752513    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752516    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752518    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752523    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752526    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752528    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752531    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.771577 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752539    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752542    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752544    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752546    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752549    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752551    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752555    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752559    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752563    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752566    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752569    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752572    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.752574    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755947    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755960    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755964    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755967    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755971    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755975    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755977    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.772206 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755980    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755983    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755985    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755988    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755990    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755993    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.755996    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756003    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756006    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756008    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756011    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756014    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756018    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756020    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756023    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756026    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756029    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756032    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756034    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756038    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.773032 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756040    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756043    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756045    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756048    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756050    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756053    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756055    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756058    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756061    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756063    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756066    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756069    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756071    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756076    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756080    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756083    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756086    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756088    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756091    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.773700 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756095    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756097    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756100    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756103    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756105    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756108    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756111    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756115    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756119    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756123    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756126    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756129    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756132    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756135    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756138    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756141    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756144    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756148    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756152    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.774327 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756155    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756158    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756161    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756164    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756166    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756169    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756171    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756174    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756176    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756180    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756183    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756186    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756189    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756192    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756195    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756198    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756201    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756203    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756213    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756215    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.774860 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756218    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756221    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756225    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756227    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756230    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756233    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756236    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756238    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756241    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756243    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756246    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756249    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756251    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756254    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756703    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756710    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756714    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756718    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756721    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.775481 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756725    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756728    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756731    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756734    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756737    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756740    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756744    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756746    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756749    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756751    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756754    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756757    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756760    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756762    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756765    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756768    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756770    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756773    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756776    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756778    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.776215 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756796    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756799    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756802    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756804    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756807    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756810    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756812    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756815    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756818    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756820    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756823    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756825    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756828    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756830    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756833    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756835    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756838    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756840    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756844    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.777218 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756847    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756849    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756852    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756855    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756857    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756860    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756869    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756871    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756874    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756876    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756879    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756882    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756884    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756887    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756889    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756892    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756895    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756898    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756901    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756903    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.778077 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756906    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756909    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756911    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756914    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756916    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756919    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756921    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756924    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756926    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756929    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756931    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756935    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756939    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756942    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756945    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756947    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756950    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756953    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756956    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756959    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.778725 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756962    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756965    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756968    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756970    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756973    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756977    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756980    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756983    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756986    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756989    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756992    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756994    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756997    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.756999    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757002    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757074    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757081    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757087    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757099    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757103    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757110    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 16:23:21.779355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757115    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757119    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757124    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757126    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757130    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757133    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757136    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757139    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757142    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757145    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757149    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757152    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757156    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757159    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757162    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757165    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757168    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757172    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757175    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757178    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757182    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757184    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757187    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757191    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 16:23:21.779986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757194    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757197    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757201    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757204    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757208    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757210    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757213    2565 flags.go:64] FLAG: --enable-server="true"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757217    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757222    2565 flags.go:64] FLAG: --event-burst="100"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757225    2565 flags.go:64] FLAG: --event-qps="50"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757229    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757232    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757235    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757239    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757242    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757245    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757248    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757251    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757254    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757257    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757260    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757263    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757266    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757268    2565 flags.go:64] FLAG: --feature-gates=""
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757272    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 16:23:21.780644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757275    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757278    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757281    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757284    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757287    2565 flags.go:64] FLAG: --help="false"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757290    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-140-28.ec2.internal"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757293    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757296    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757298    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757301    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757305    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757308    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757311    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757314    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757320    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757323    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757326    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757331    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757333    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757336    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757339    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757342    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757345    2565 flags.go:64] FLAG: --lock-file=""
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757347    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 16:23:21.781501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757350    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757353    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757363    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757367    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757370    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757373    2565 flags.go:64] FLAG: --logging-format="text"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757375    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757379    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757382    2565 flags.go:64] FLAG: --manifest-url=""
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757384    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757389    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757392    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757396    2565 flags.go:64] FLAG: --max-pods="110"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757399    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757402    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757405    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757407    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757410    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757413    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757416    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757423    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757426    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757430    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 16:23:21.784306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757433    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757436    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757441    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757445    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757449    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757452    2565 flags.go:64] FLAG: --port="10250"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757454    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757457    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-005459c1e2fa40de5"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757460    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757463    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757466    2565 flags.go:64] FLAG: --register-node="true"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757469    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757473    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757476    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757479    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757482    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757486    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757488    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757491    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757494    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757497    2565 flags.go:64] FLAG: --runonce="false"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757500    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757503    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757506    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757509    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 16:23:21.862091 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757512    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 16:23:21.844315 ip-10-0-140-28 systemd[1]: Started Kubernetes Kubelet.
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757515    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757518    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757521    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757523    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757526    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757530    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757533    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757536    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757539    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757543    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757548    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757551    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757554    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757558    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757561    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757564    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757567    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757569    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757572    2565 flags.go:64] FLAG: --v="2"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757576    2565 flags.go:64] FLAG: --version="false"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757581    2565 flags.go:64] FLAG: --vmodule=""
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757585    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.757588    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757710    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.866538 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757714    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757717    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757720    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757724    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757728    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757731    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757734    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757737    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757740    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757743    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757746    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757749    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757751    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757755    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757758    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757760    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757762    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757766    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757769    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.867749 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757771    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757774    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757776    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757779    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757794    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757797    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757799    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757802    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757805    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757807    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757813    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757816    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757818    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757821    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757824    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757826    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757829    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757831    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757834    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757836    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.868649 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757839    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757841    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757844    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757846    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757849    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757851    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757855    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757857    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757860    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757862    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757866    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757869    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757872    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757874    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757877    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757879    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757882    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757884    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757888    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757892    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.869766 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757895    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757897    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757900    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757904    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757906    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757909    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757912    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757914    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757917    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757919    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757922    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757925    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757927    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757929    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757932    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757936    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757940    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757942    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757947    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.870632 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757950    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757952    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757955    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757958    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757961    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757963    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757966    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757968    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757971    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757973    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757976    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757978    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757981    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757983    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757986    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757989    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757992    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757996    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.757999    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.871387 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.758001    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.759298    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770118    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770136    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770153    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770161    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770230    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770235    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770239    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770242    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770245    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770248    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770251    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770254    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770257    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.872149 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770260    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770263    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770268    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770270    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770275    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770279    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770283    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770286    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770289    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770292    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770294    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770297    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770300    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770303    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770305    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770308    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770311    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770314    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770316    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.872696 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770319    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770324    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770328    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770331    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770334    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770336    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770339    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770342    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770345    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770347    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770350    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770353    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770356    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770359    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770361    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770367    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770371    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770374    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770377    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.873438 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770379    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770382    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770384    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770387    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770389    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770392    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770394    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770397    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770400    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770402    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770405    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770407    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770410    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770412    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770415    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770418    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770421    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770423    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770426    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770428    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.874182 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770431    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770433    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770436    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770438    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770441    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770443    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770446    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770448    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770451    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770453    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770456    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770459    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770462    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770464    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770467    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770469    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770472    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770475    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770477    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770480    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.874953 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770483    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770486    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770488    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770491    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770493    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770496    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770498    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770501    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770504    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770507    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770509    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770512    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770517    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770641    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:23:21.875873 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770647    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770650    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770653    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770659    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770662    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770664    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770667    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770670    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770673    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770676    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770679    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770682    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770684    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770688    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770691    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770695    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770699    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770703    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770706    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770709    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:23:21.876280 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770711    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770714    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770717    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770720    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770723    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770725    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770728    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770731    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770734    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770736    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770739    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770741    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770744    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770747    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770750    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770754    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770756    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770759    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770763    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:23:21.876983 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770767    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770769    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770772    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770775    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770777    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770780    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770798    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770801    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770804    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770806    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770809    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770812    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770814    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770817    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770820    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770823    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770825    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770828    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770830    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770833    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:23:21.877582 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770835    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770838    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770841    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770844    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770846    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770851    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770854    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770856    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770860    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770863    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770865    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770868    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770871    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770873    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770875    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770878    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770881    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770884    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770886    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:23:21.878171 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770889    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770892    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770894    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770897    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770899    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770902    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770905    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770907    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770910    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770913    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770915    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770917    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770920    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770923    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770925    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770929    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770931    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770935    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770940    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:23:21.878687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:21.770943    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.770948    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.771097    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.773349    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.774771    2565 server.go:1019] "Starting client certificate rotation"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.774896    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.775706    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.778347    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.782807    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.805689    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.808127    2565 log.go:25] "Validated CRI v1 image API"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.809608    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 16:23:21.879299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.811721    2565 fs.go:135] Filesystem UUIDs: map[1f082078-b974-4ced-99cf-4cbb50510f55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8f67a5ce-5dd4-4498-846e-eaaf257261b9:/dev/nvme0n1p3]
Jul 07 16:23:21.879662 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.811735    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.819067    2565 manager.go:216] Machine: {Timestamp:2026-07-07 16:23:21.817652 +0000 UTC m=+0.350568652 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101707 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2248376d8f576b4ef45c9631cda377 SystemUUID:ec224837-6d8f-576b-4ef4-5c9631cda377 BootID:7a33ac0c-99ea-4f21-8941-b80506177012 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b3:3f:c1:da:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b3:3f:c1:da:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:de:b7:ba:ab:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.819184    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.819264    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.822030    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.822055    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.822796    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.822805    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 16:23:21.879806 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.822813    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.823457    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.824621    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.824661    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827340    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827457    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827484    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827497    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827514    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827524    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827531    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827899    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.827912    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.837280    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.839015    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.839040    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840490    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840502    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840508    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840514    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840529    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840533    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840541    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840545    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840551    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840557    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840564    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840579    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840613    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.840620    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.843001    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.843048    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.843080    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.843546    2565 server.go:1268] "Started kubelet"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.843595    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.843742    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.843828    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.845366    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 16:23:22.009934 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.846675    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.850365    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-28.ec2.internal.18c00e81829b5231  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-28.ec2.internal,UID:ip-10-0-140-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-28.ec2.internal,},FirstTimestamp:2026-07-07 16:23:21.843503665 +0000 UTC m=+0.376420298,LastTimestamp:2026-07-07 16:23:21.843503665 +0000 UTC m=+0.376420298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-28.ec2.internal,}"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.852427    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.852439    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.852501    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853061    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853063    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853103    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853186    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853195    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853860    2565 factory.go:55] Registering systemd factory
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.853893    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854106    2565 factory.go:153] Registering CRI-O factory
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854120    2565 factory.go:223] Registration of the crio container factory successfully
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854165    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854187    2565 factory.go:103] Registering Raw factory
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854203    2565 manager.go:1196] Started watching for new ooms in manager
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.854577    2565 manager.go:349] Starting recovery of all containers
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.858124    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.858194    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.865575    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b5pfj"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.870204    2565 manager.go:354] Recovery completed
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.874557    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b5pfj"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.878694    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.880543    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.880571    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.880584    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881107    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881113    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881118    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881134    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881899    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.034262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881922    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.881931    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.882183    2565 policy_none.go:49] "None policy: Start"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.882196    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.882205    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.883056    2565 policy_none.go:47] "Start"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.883068    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-28.ec2.internal.18c00e8184d0ac3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-28.ec2.internal,UID:ip-10-0-140-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-28.ec2.internal,},FirstTimestamp:2026-07-07 16:23:21.880554556 +0000 UTC m=+0.413471197,LastTimestamp:2026-07-07 16:23:21.880554556 +0000 UTC m=+0.413471197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-28.ec2.internal,}"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920388    2565 manager.go:337] "Starting Device Plugin manager"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.920551    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920565    2565 server.go:87] "Starting device plugin registration server"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920824    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920834    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920912    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.920994    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:21.921005    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.921533    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:21.921587    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.020909    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.022416    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.022442    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.022453    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.022479    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.032192    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.035625 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.032213    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-28.ec2.internal\": node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.051390    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.087683    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.088720    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.088763    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.088810    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.088847    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 16:23:22.148924 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.092366    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:23:22.153354 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.152341    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.189674    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"]
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.189765    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.193700    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.193728    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.193739    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195082    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195214    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195751    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195757    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195800    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195813    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195816    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.195833    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.197398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.197425    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.198091    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.198118    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.198132    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.229379    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-28.ec2.internal\" not found" node="ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.238522 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.232646    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-28.ec2.internal\" not found" node="ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.322235 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.252880    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.322235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.255054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6eb5e9c0a661aaef32578bdf54d1da55-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6eb5e9c0a661aaef32578bdf54d1da55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.322235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.255077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.322235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.255095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.353362    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6eb5e9c0a661aaef32578bdf54d1da55-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6eb5e9c0a661aaef32578bdf54d1da55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355633    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355667    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f9260fbdb911f043b0adc8e0f03aaafe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"f9260fbdb911f043b0adc8e0f03aaafe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.421546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.355662    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6eb5e9c0a661aaef32578bdf54d1da55-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6eb5e9c0a661aaef32578bdf54d1da55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.549010 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.453759    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.549010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.532025    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.549010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.535496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:22.664333 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.554524    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.664333 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.654954    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.772898 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.755382    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.774998 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.774632    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 16:23:22.774998 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.774762    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 16:23:22.864971 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.856118    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:22.974973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.879029    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:18:21 +0000 UTC" deadline="2028-01-14 05:09:26.121031076 +0000 UTC"
Jul 07 16:23:22.974973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.879053    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13332h46m3.241980962s"
Jul 07 16:23:22.974973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:22.924243    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:23:22.974973 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:22.956446    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:23.081053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.005417    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:23:23.081053 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.057069    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found"
Jul 07 16:23:23.081053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.073709    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:23:23.153058 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.153028    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:23.168865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.168851    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:23:23.169865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.169854    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"
Jul 07 16:23:23.173390 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:23.173354    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9260fbdb911f043b0adc8e0f03aaafe.slice/crio-26a10489224cc4a5aadc7c06e23a35eaeeab98db170e21624bdfc63c76608fd7 WatchSource:0}: Error finding container 26a10489224cc4a5aadc7c06e23a35eaeeab98db170e21624bdfc63c76608fd7: Status 404 returned error can't find the container with id 26a10489224cc4a5aadc7c06e23a35eaeeab98db170e21624bdfc63c76608fd7
Jul 07 16:23:23.177278 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.177258    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:23:23.186412 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.186397    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:23:23.371889 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:23.371850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb5e9c0a661aaef32578bdf54d1da55.slice/crio-e792a36c5be08ca5f44e2bf450c10b62efe3622ff068fe17b2a2f850c932ccde WatchSource:0}: Error finding container e792a36c5be08ca5f44e2bf450c10b62efe3622ff068fe17b2a2f850c932ccde: Status 404 returned error can't find the container with id e792a36c5be08ca5f44e2bf450c10b62efe3622ff068fe17b2a2f850c932ccde
Jul 07 16:23:23.774200 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.774130    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:23:23.828560 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.828535    2565 apiserver.go:52] "Watching apiserver"
Jul 07 16:23:23.839177 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.838990    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 16:23:23.840390 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.840359    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-c62q6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal","openshift-multus/network-metrics-daemon-p8h5z","openshift-network-diagnostics/network-check-target-bvnp6","openshift-network-operator/iptables-alerter-99ln6","openshift-ovn-kubernetes/ovnkube-node-wm47z","kube-system/global-pull-secret-syncer-6rrzf","kube-system/konnectivity-agent-tw2zg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq","openshift-cluster-node-tuning-operator/tuned-prxkt","openshift-multus/multus-additional-cni-plugins-kr42w","openshift-multus/multus-bwspc","kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal","openshift-dns/node-resolver-fn2kg"]
Jul 07 16:23:23.841985 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.841962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.842102 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.842052    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:23.843903 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.843449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.845235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.845216    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.845328 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.845290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:23.847166 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.847076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:23.847166 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.847132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:23.848474 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.848225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.850210 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.849861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.851076 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.851058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.852384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.851511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 16:23:23.852384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.851793    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.852687 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.852670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.852993 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.852975    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 16:23:23.854110 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.853555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.854110 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.853895    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-745zh\""
Jul 07 16:23:23.854110 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.853901    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.854320 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854498    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb54p\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854549    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854645    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854901    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 16:23:23.855041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.854987    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.855650    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rxp2x\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.855739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.855866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qdj57\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.855990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856253    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856295    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856416    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856502    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856646    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5lwz9\""
Jul 07 16:23:23.857180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.856682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.857937 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.857855    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.859154 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.859133    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:23.861139 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.860485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-var-lib-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-dbus\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-cnibin\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249ad43d-fc4a-4a80-972b-e88b626b20fa-tmp-dir\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.861985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mspw\" (UniqueName: \"kubernetes.io/projected/ac619709-6f42-4267-9cee-72489b234fce-kube-api-access-9mspw\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-sys-fs\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b23933-59e6-4828-9915-724175f870ca-agent-certs\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-ovn\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-log-socket\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-config\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.862516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b23933-59e6-4828-9915-724175f870ca-konnectivity-ca\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-slash\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-netns\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-registration-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-device-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdz8\" (UniqueName: \"kubernetes.io/projected/ababdc3c-82b9-467c-91bd-c6464619cb2c-kube-api-access-fbdz8\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42c85b9b-6749-4c66-bf67-370af90c338a-iptables-alerter-script\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-bin\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kfx\" (UniqueName: \"kubernetes.io/projected/50299ae9-ceb8-450e-9854-159da1baf43e-kube-api-access-62kfx\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42c85b9b-6749-4c66-bf67-370af90c338a-host-slash\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249ad43d-fc4a-4a80-972b-e88b626b20fa-hosts-file\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.863186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7w7\" (UniqueName: \"kubernetes.io/projected/249ad43d-fc4a-4a80-972b-e88b626b20fa-kube-api-access-vk7w7\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-systemd-units\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.862996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-node-log\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-netd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-env-overrides\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-script-lib\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-kubelet\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-etc-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.863738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-socket-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sw97\" (UniqueName: \"kubernetes.io/projected/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kube-api-access-7sw97\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac619709-6f42-4267-9cee-72489b234fce-ovn-node-metrics-cert\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-kubelet-config\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-system-cni-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-os-release\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8lb\" (UniqueName: \"kubernetes.io/projected/42c85b9b-6749-4c66-bf67-370af90c338a-kube-api-access-5d8lb\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.864294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.863521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-systemd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.864671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.864401    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 16:23:23.864671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.864400    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.864671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.864612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.864848 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.864740    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 16:23:23.865088 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.864955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 16:23:23.865252 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865232    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 16:23:23.865473 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865450    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 16:23:23.865549 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865495    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wf9lr\""
Jul 07 16:23:23.865549 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7bmlt\""
Jul 07 16:23:23.865644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 16:23:23.865958 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.865939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gvp8m\""
Jul 07 16:23:23.866270 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.866116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lbgbr\""
Jul 07 16:23:23.871855 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.871837    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:23:23.911668 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.911648    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8mjv"
Jul 07 16:23:23.919527 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.919503    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8mjv"
Jul 07 16:23:23.954028 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.953959    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 16:23:23.964793 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sw97\" (UniqueName: \"kubernetes.io/projected/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kube-api-access-7sw97\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-cnibin\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964854    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-daemon-config\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964917    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-cnibin\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-kubelet\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.964978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.964987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-socket-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46480d98-8a3a-4a69-8214-db6d53f2f9d3-serviceca\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-system-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac619709-6f42-4267-9cee-72489b234fce-ovn-node-metrics-cert\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-kubelet-config\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-system-cni-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46480d98-8a3a-4a69-8214-db6d53f2f9d3-host\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-socket-dir-parent\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmj6\" (UniqueName: \"kubernetes.io/projected/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-kube-api-access-7tmj6\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-systemd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965299    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-socket-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965340    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-kubelet\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.965594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965361    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-systemd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965411    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965429    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965490    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-kubelet-config\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965528    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-system-cni-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.965556    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965624    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.965672    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:24.465652425 +0000 UTC m=+2.998569061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965674    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mspw\" (UniqueName: \"kubernetes.io/projected/ac619709-6f42-4267-9cee-72489b234fce-kube-api-access-9mspw\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965746    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-host\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.966225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-tuned\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-k8s-cni-cncf-io\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-hostroot\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-ovn\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-config\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.965998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-netns\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-registration-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-device-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966061    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47w5k\" (UniqueName: \"kubernetes.io/projected/a485ff9d-efde-4b4b-949a-27992ce276e0-kube-api-access-47w5k\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-bin\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966130    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-registration-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kfx\" (UniqueName: \"kubernetes.io/projected/50299ae9-ceb8-450e-9854-159da1baf43e-kube-api-access-62kfx\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966134    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-run-ovn\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.966914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966149    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-bin\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42c85b9b-6749-4c66-bf67-370af90c338a-host-slash\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966260    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-device-dir\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966258    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-kubelet\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966320    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42c85b9b-6749-4c66-bf67-370af90c338a-host-slash\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249ad43d-fc4a-4a80-972b-e88b626b20fa-hosts-file\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966368    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249ad43d-fc4a-4a80-972b-e88b626b20fa-hosts-file\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-systemd-units\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-node-log\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-netd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-env-overrides\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966461    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-systemd-units\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-script-lib\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966479    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-node-log\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.967563 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8lb\" (UniqueName: \"kubernetes.io/projected/42c85b9b-6749-4c66-bf67-370af90c338a-kube-api-access-5d8lb\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966504    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-cni-netd\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-dbus\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-etc-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-config\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42c85b9b-6749-4c66-bf67-370af90c338a-iptables-alerter-script\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-var-lib-kubelet\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-multus\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966685    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee74c68-b023-4f56-8591-8449f020992f-dbus\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-os-release\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-modprobe-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966735    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50299ae9-ceb8-450e-9854-159da1baf43e-os-release\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.968277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966831    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-env-overrides\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966843    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966881    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-etc-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-bin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-var-lib-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.966977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-run\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cnibin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967004    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-var-lib-openvswitch\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967031    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac619709-6f42-4267-9cee-72489b234fce-ovnkube-script-lib\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-os-release\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-conf-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967089    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42c85b9b-6749-4c66-bf67-370af90c338a-iptables-alerter-script\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.968901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249ad43d-fc4a-4a80-972b-e88b626b20fa-tmp-dir\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-sys-fs\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysconfig\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-sys\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967193    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-sys-fs\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-lib-modules\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-tmp\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967240    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-etc-kubernetes\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b23933-59e6-4828-9915-724175f870ca-agent-certs\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-log-socket\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967311    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50299ae9-ceb8-450e-9854-159da1baf43e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-kubernetes\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967353    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-log-socket\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b23933-59e6-4828-9915-724175f870ca-konnectivity-ca\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.969493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967388    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249ad43d-fc4a-4a80-972b-e88b626b20fa-tmp-dir\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-slash\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-netns\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967471    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-slash\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdz8\" (UniqueName: \"kubernetes.io/projected/ababdc3c-82b9-467c-91bd-c6464619cb2c-kube-api-access-fbdz8\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-multus-certs\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967512    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac619709-6f42-4267-9cee-72489b234fce-host-run-netns\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-conf\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-systemd\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.967609    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49g7\" (UniqueName: \"kubernetes.io/projected/46480d98-8a3a-4a69-8214-db6d53f2f9d3-kube-api-access-n49g7\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.967690    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:24.467675129 +0000 UTC m=+3.000591763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cni-binary-copy\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:23.970248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7w7\" (UniqueName: \"kubernetes.io/projected/249ad43d-fc4a-4a80-972b-e88b626b20fa-kube-api-access-vk7w7\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.970921 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.967859    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b23933-59e6-4828-9915-724175f870ca-konnectivity-ca\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.970921 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.969391    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac619709-6f42-4267-9cee-72489b234fce-ovn-node-metrics-cert\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.973666 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.973644    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b23933-59e6-4828-9915-724175f870ca-agent-certs\") pod \"konnectivity-agent-tw2zg\" (UID: \"18b23933-59e6-4828-9915-724175f870ca\") " pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:23.977298 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.977197    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:23.977298 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.977216    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:23.977298 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.977226    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:23.977298 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:23.977278    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:24.477263183 +0000 UTC m=+3.010179818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:23.979663 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.979593    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8lb\" (UniqueName: \"kubernetes.io/projected/42c85b9b-6749-4c66-bf67-370af90c338a-kube-api-access-5d8lb\") pod \"iptables-alerter-99ln6\" (UID: \"42c85b9b-6749-4c66-bf67-370af90c338a\") " pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:23.983506 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.983486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7w7\" (UniqueName: \"kubernetes.io/projected/249ad43d-fc4a-4a80-972b-e88b626b20fa-kube-api-access-vk7w7\") pod \"node-resolver-fn2kg\" (UID: \"249ad43d-fc4a-4a80-972b-e88b626b20fa\") " pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:23.985187 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.985166    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mspw\" (UniqueName: \"kubernetes.io/projected/ac619709-6f42-4267-9cee-72489b234fce-kube-api-access-9mspw\") pod \"ovnkube-node-wm47z\" (UID: \"ac619709-6f42-4267-9cee-72489b234fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:23.987507 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.987465    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdz8\" (UniqueName: \"kubernetes.io/projected/ababdc3c-82b9-467c-91bd-c6464619cb2c-kube-api-access-fbdz8\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:23.988527 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.988442    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kfx\" (UniqueName: \"kubernetes.io/projected/50299ae9-ceb8-450e-9854-159da1baf43e-kube-api-access-62kfx\") pod \"multus-additional-cni-plugins-kr42w\" (UID: \"50299ae9-ceb8-450e-9854-159da1baf43e\") " pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:23.990886 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:23.990866    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sw97\" (UniqueName: \"kubernetes.io/projected/6b60ed0b-dbd8-48c5-9309-bc83f538afe4-kube-api-access-7sw97\") pod \"aws-ebs-csi-driver-node-6b5zq\" (UID: \"6b60ed0b-dbd8-48c5-9309-bc83f538afe4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-k8s-cni-cncf-io\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-hostroot\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-netns\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47w5k\" (UniqueName: \"kubernetes.io/projected/a485ff9d-efde-4b4b-949a-27992ce276e0-kube-api-access-47w5k\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-kubelet\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-var-lib-kubelet\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-multus\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-modprobe-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-bin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-run\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cnibin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-os-release\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.069889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-conf-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068927    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-multus\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysconfig\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-sys\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068983    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-k8s-cni-cncf-io\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.068990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-lib-modules\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-tmp\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069017    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-hostroot\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-etc-kubernetes\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069051    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-netns\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-kubernetes\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-multus-certs\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-conf\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-systemd\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49g7\" (UniqueName: \"kubernetes.io/projected/46480d98-8a3a-4a69-8214-db6d53f2f9d3-kube-api-access-n49g7\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.070631 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cni-binary-copy\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-daemon-config\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46480d98-8a3a-4a69-8214-db6d53f2f9d3-serviceca\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069302    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-kubelet\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-system-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46480d98-8a3a-4a69-8214-db6d53f2f9d3-host\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069358    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-var-lib-kubelet\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-socket-dir-parent\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmj6\" (UniqueName: \"kubernetes.io/projected/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-kube-api-access-7tmj6\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-host\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-tuned\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069590    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069640    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-var-lib-cni-bin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069733    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-run\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.071366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069797    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cnibin\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-os-release\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069926    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-conf-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069989    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysconfig\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.070027    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-sys\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.070529    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-daemon-config\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071172    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46480d98-8a3a-4a69-8214-db6d53f2f9d3-serviceca\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071252    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-system-cni-dir\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071295    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46480d98-8a3a-4a69-8214-db6d53f2f9d3-host\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071342    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-multus-socket-dir-parent\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071480    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-sysctl-conf\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071527    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-etc-kubernetes\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071571    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-kubernetes\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071611    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-host-run-multus-certs\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.071667    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-host\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072590 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.072102    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-systemd\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072590 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.072497    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-cni-binary-copy\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.072590 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.069493    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-lib-modules\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.072723 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.072614    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-modprobe-d\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.073557 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.073516    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-etc-tuned\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.074812 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.074793    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a485ff9d-efde-4b4b-949a-27992ce276e0-tmp\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.083597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.083560    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47w5k\" (UniqueName: \"kubernetes.io/projected/a485ff9d-efde-4b4b-949a-27992ce276e0-kube-api-access-47w5k\") pod \"tuned-prxkt\" (UID: \"a485ff9d-efde-4b4b-949a-27992ce276e0\") " pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.097835 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.097679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" event={"ID":"6eb5e9c0a661aaef32578bdf54d1da55","Type":"ContainerStarted","Data":"e792a36c5be08ca5f44e2bf450c10b62efe3622ff068fe17b2a2f850c932ccde"}
Jul 07 16:23:24.100031 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.100006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"f9260fbdb911f043b0adc8e0f03aaafe","Type":"ContainerStarted","Data":"26a10489224cc4a5aadc7c06e23a35eaeeab98db170e21624bdfc63c76608fd7"}
Jul 07 16:23:24.101035 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.100989    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmj6\" (UniqueName: \"kubernetes.io/projected/2f711f0e-5198-4415-a6fe-63ff5cfb9b1d-kube-api-access-7tmj6\") pod \"multus-bwspc\" (UID: \"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d\") " pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.102303 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.102261    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49g7\" (UniqueName: \"kubernetes.io/projected/46480d98-8a3a-4a69-8214-db6d53f2f9d3-kube-api-access-n49g7\") pod \"node-ca-c62q6\" (UID: \"46480d98-8a3a-4a69-8214-db6d53f2f9d3\") " pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.156032 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.155917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kr42w"
Jul 07 16:23:24.166241 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.166207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99ln6"
Jul 07 16:23:24.178352 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.178332    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:24.184054 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.184034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq"
Jul 07 16:23:24.191109 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.191087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fn2kg"
Jul 07 16:23:24.198862 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.198833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:24.205406 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.205388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prxkt"
Jul 07 16:23:24.215144 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.215125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c62q6"
Jul 07 16:23:24.220371 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.220345    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwspc"
Jul 07 16:23:24.472578 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.472498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:24.472578 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.472562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:24.472807 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.472662    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:24.472807 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.472735    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.472716939 +0000 UTC m=+4.005633573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:24.472807 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.472664    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:24.472953 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.472823    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.472806081 +0000 UTC m=+4.005722705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:24.573740 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.573706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:24.573904 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.573892    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:24.573940 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.573913    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:24.573940 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.573924    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:24.574015 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:24.573978    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.573964346 +0000 UTC m=+4.106880987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:24.921874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.921838    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:18:23 +0000 UTC" deadline="2028-02-06 16:29:09.831848826 +0000 UTC"
Jul 07 16:23:24.921874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:24.921865    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13896h5m44.909986908s"
Jul 07 16:23:24.989721 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.989698    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac619709_6f42_4267_9cee_72489b234fce.slice/crio-26c3bc1e2bc28e5f642aaa110c705272545b3ecb5d0629b0af54f07912da86ec WatchSource:0}: Error finding container 26c3bc1e2bc28e5f642aaa110c705272545b3ecb5d0629b0af54f07912da86ec: Status 404 returned error can't find the container with id 26c3bc1e2bc28e5f642aaa110c705272545b3ecb5d0629b0af54f07912da86ec
Jul 07 16:23:24.993614 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.993584    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b23933_59e6_4828_9915_724175f870ca.slice/crio-93891e561ecfe6f4f1dad1617df4909c47853bd311f8e308685c60d0ba69c777 WatchSource:0}: Error finding container 93891e561ecfe6f4f1dad1617df4909c47853bd311f8e308685c60d0ba69c777: Status 404 returned error can't find the container with id 93891e561ecfe6f4f1dad1617df4909c47853bd311f8e308685c60d0ba69c777
Jul 07 16:23:24.995736 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.995715    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46480d98_8a3a_4a69_8214_db6d53f2f9d3.slice/crio-cbd83be8869c6bd715da5b1f184be7a0d596d118f9eaeb69dc1e93d1c7660851 WatchSource:0}: Error finding container cbd83be8869c6bd715da5b1f184be7a0d596d118f9eaeb69dc1e93d1c7660851: Status 404 returned error can't find the container with id cbd83be8869c6bd715da5b1f184be7a0d596d118f9eaeb69dc1e93d1c7660851
Jul 07 16:23:24.996554 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.996533    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda485ff9d_efde_4b4b_949a_27992ce276e0.slice/crio-f5799419d4943cbb7a5600f5c25ea9a2b0785f9ddd56a15c1bbdef2e9b0a5e2a WatchSource:0}: Error finding container f5799419d4943cbb7a5600f5c25ea9a2b0785f9ddd56a15c1bbdef2e9b0a5e2a: Status 404 returned error can't find the container with id f5799419d4943cbb7a5600f5c25ea9a2b0785f9ddd56a15c1bbdef2e9b0a5e2a
Jul 07 16:23:24.997951 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.997877    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249ad43d_fc4a_4a80_972b_e88b626b20fa.slice/crio-0663ec474159bba16d75053c627b6184f2bd08c4121e9ef16e4710981cf3dd0c WatchSource:0}: Error finding container 0663ec474159bba16d75053c627b6184f2bd08c4121e9ef16e4710981cf3dd0c: Status 404 returned error can't find the container with id 0663ec474159bba16d75053c627b6184f2bd08c4121e9ef16e4710981cf3dd0c
Jul 07 16:23:24.998675 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.998650    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b60ed0b_dbd8_48c5_9309_bc83f538afe4.slice/crio-943cc621c5178a617df85abf4e09566cee7a7e82518666228de0768e3339df04 WatchSource:0}: Error finding container 943cc621c5178a617df85abf4e09566cee7a7e82518666228de0768e3339df04: Status 404 returned error can't find the container with id 943cc621c5178a617df85abf4e09566cee7a7e82518666228de0768e3339df04
Jul 07 16:23:24.999436 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:24.999337    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c85b9b_6749_4c66_bf67_370af90c338a.slice/crio-fd4bdde6deba7cbd16360bdd23cdb3b2eac04eb856cf600e98937870051fbe57 WatchSource:0}: Error finding container fd4bdde6deba7cbd16360bdd23cdb3b2eac04eb856cf600e98937870051fbe57: Status 404 returned error can't find the container with id fd4bdde6deba7cbd16360bdd23cdb3b2eac04eb856cf600e98937870051fbe57
Jul 07 16:23:25.000306 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:25.000171    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50299ae9_ceb8_450e_9854_159da1baf43e.slice/crio-95eacdee99e00168910fceb1d658dda9fb415328e36d6bd55f3b994dc89d8ede WatchSource:0}: Error finding container 95eacdee99e00168910fceb1d658dda9fb415328e36d6bd55f3b994dc89d8ede: Status 404 returned error can't find the container with id 95eacdee99e00168910fceb1d658dda9fb415328e36d6bd55f3b994dc89d8ede
Jul 07 16:23:25.002262 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:25.002239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f711f0e_5198_4415_a6fe_63ff5cfb9b1d.slice/crio-19d7152b1def63543d95128273a9e55e3cfe4c22702caf02b6555f09b64f49e3 WatchSource:0}: Error finding container 19d7152b1def63543d95128273a9e55e3cfe4c22702caf02b6555f09b64f49e3: Status 404 returned error can't find the container with id 19d7152b1def63543d95128273a9e55e3cfe4c22702caf02b6555f09b64f49e3
Jul 07 16:23:25.103937 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.103769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" event={"ID":"6eb5e9c0a661aaef32578bdf54d1da55","Type":"ContainerStarted","Data":"e9a120ac3d0540abdd086fdbcb752ff4c7815fc6b25c48c91c26e0c00ece7069"}
Jul 07 16:23:25.104869 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.104843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerStarted","Data":"95eacdee99e00168910fceb1d658dda9fb415328e36d6bd55f3b994dc89d8ede"}
Jul 07 16:23:25.105953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.105909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99ln6" event={"ID":"42c85b9b-6749-4c66-bf67-370af90c338a","Type":"ContainerStarted","Data":"fd4bdde6deba7cbd16360bdd23cdb3b2eac04eb856cf600e98937870051fbe57"}
Jul 07 16:23:25.106973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.106950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq" event={"ID":"6b60ed0b-dbd8-48c5-9309-bc83f538afe4","Type":"ContainerStarted","Data":"943cc621c5178a617df85abf4e09566cee7a7e82518666228de0768e3339df04"}
Jul 07 16:23:25.108173 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.108157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fn2kg" event={"ID":"249ad43d-fc4a-4a80-972b-e88b626b20fa","Type":"ContainerStarted","Data":"0663ec474159bba16d75053c627b6184f2bd08c4121e9ef16e4710981cf3dd0c"}
Jul 07 16:23:25.109218 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.109198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prxkt" event={"ID":"a485ff9d-efde-4b4b-949a-27992ce276e0","Type":"ContainerStarted","Data":"f5799419d4943cbb7a5600f5c25ea9a2b0785f9ddd56a15c1bbdef2e9b0a5e2a"}
Jul 07 16:23:25.110157 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.110136    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tw2zg" event={"ID":"18b23933-59e6-4828-9915-724175f870ca","Type":"ContainerStarted","Data":"93891e561ecfe6f4f1dad1617df4909c47853bd311f8e308685c60d0ba69c777"}
Jul 07 16:23:25.111336 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.111312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwspc" event={"ID":"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d","Type":"ContainerStarted","Data":"19d7152b1def63543d95128273a9e55e3cfe4c22702caf02b6555f09b64f49e3"}
Jul 07 16:23:25.112314 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.112292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c62q6" event={"ID":"46480d98-8a3a-4a69-8214-db6d53f2f9d3","Type":"ContainerStarted","Data":"cbd83be8869c6bd715da5b1f184be7a0d596d118f9eaeb69dc1e93d1c7660851"}
Jul 07 16:23:25.113334 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.113317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"26c3bc1e2bc28e5f642aaa110c705272545b3ecb5d0629b0af54f07912da86ec"}
Jul 07 16:23:25.481247 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.480985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:25.481247 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.481044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:25.481247 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.481150    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:25.481247 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.481201    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:27.481186916 +0000 UTC m=+6.014103553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:25.481516 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.481150    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:25.481563 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.481535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:27.481519149 +0000 UTC m=+6.014435773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:25.582370 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.582263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:25.582504 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.582433    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:25.582504 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.582454    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:25.582504 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.582466    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:25.582647 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:25.582547    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:27.582527911 +0000 UTC m=+6.115444546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:25.922167 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.922092    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:18:23 +0000 UTC" deadline="2028-03-24 01:52:28.733507462 +0000 UTC"
Jul 07 16:23:25.922167 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:25.922148    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15009h29m2.811362939s"
Jul 07 16:23:26.092185 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.092157    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:26.092340 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:26.092287    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:26.092651 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.092636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:26.092767 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:26.092747    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:26.092849 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.092837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:26.092961 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:26.092941    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:26.122566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.122528    2565 generic.go:356] "Generic (PLEG): container finished" podID="f9260fbdb911f043b0adc8e0f03aaafe" containerID="8f3a717ef86d90b123416a108c5f13562a1d8c3ed4d7c22fdd1cd5425cfa3d8c" exitCode=0
Jul 07 16:23:26.122664 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.122614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"f9260fbdb911f043b0adc8e0f03aaafe","Type":"ContainerDied","Data":"8f3a717ef86d90b123416a108c5f13562a1d8c3ed4d7c22fdd1cd5425cfa3d8c"}
Jul 07 16:23:26.156854 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:26.156284    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" podStartSLOduration=4.156257177 podStartE2EDuration="4.156257177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:25.120223171 +0000 UTC m=+3.653139908" watchObservedRunningTime="2026-07-07 16:23:26.156257177 +0000 UTC m=+4.689173808"
Jul 07 16:23:27.155825 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:27.154772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"f9260fbdb911f043b0adc8e0f03aaafe","Type":"ContainerStarted","Data":"b572e1b3969d89d9721612e865648f1a112be5d04966b1c4880f9e2477298e0b"}
Jul 07 16:23:27.501534 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:27.501443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:27.501534 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:27.501510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:27.501737 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.501630    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:27.501737 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.501715    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:31.501671863 +0000 UTC m=+10.034588487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:27.502163 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.502135    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:27.502269 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.502195    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:31.502177739 +0000 UTC m=+10.035094363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:27.602639 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:27.602015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:27.602639 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.602234    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:27.602639 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.602256    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:27.602639 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.602268    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:27.602639 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:27.602321    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:31.602303419 +0000 UTC m=+10.135220044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:28.089878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:28.089899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:28.090025    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:28.090440    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:28.090488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:28.090612 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:28.090578    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:30.089703 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:30.089670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:30.090163 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:30.089864    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:30.090555 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:30.090297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:30.090555 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:30.090400    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:30.090555 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:30.090428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:30.090555 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:30.090506    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:31.534860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:31.534922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.535043    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.535095    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:39.535081736 +0000 UTC m=+18.067998382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.535399    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:31.535464 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.535435    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:39.535423446 +0000 UTC m=+18.068340082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:31.635575 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:31.635538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:31.635764 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.635710    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:31.635764 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.635727    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:31.635764 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.635748    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:31.635954 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:31.635822    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:39.635803698 +0000 UTC m=+18.168720338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:32.090396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:32.090423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:32.090538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:32.090566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:32.090613    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:32.090735 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:32.090698    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:34.090114 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:34.089990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:34.090614 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:34.090117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:34.090614 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:34.090133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:34.090614 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:34.090243    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:34.090614 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:34.090305    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:34.090614 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:34.090386    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:36.089306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:36.089270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:36.089773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:36.089272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:36.089773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:36.089430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:36.089773 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:36.089419    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:36.089773 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:36.089566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:36.089773 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:36.089668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:38.089413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:38.089381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:38.089916 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:38.089381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:38.089916 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:38.089520    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:38.089916 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:38.089386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:38.089916 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:38.089598    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:38.089916 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:38.089741    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:39.595760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:39.595879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.596055    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.596163    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.596619    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:55.596585229 +0000 UTC m=+34.129501867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:39.598138 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.597115    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:23:55.597088691 +0000 UTC m=+34.130005329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:39.696385 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:39.696354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:39.696545 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.696531    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:39.696619 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.696554    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:39.696619 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.696567    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:39.696721 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:39.696625    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:23:55.696608633 +0000 UTC m=+34.229525269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:40.089289 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:40.089249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:40.089458 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:40.089261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:40.089458 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:40.089356    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:40.089458 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:40.089389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:40.089614 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:40.089503    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:40.089684 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:40.089606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:42.090215 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:42.090183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:42.090624 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:42.090298    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:42.090624 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:42.090429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:42.090624 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:42.090465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:42.090624 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:42.090555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:42.090740 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:42.090637    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:44.091819 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.091607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:44.092579 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:44.091919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:44.092579 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.091703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:44.092579 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:44.092023    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:44.092579 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.091655    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:44.092579 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:44.092099    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:44.191855 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.191305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerStarted","Data":"de93c71f611472e5403f87f84a16bd762785dc5a99ca3f49c0efbbb08cb24705"}
Jul 07 16:23:44.196591 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.196561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq" event={"ID":"6b60ed0b-dbd8-48c5-9309-bc83f538afe4","Type":"ContainerStarted","Data":"015ca12ebd26857041543f5f8f61a00a55fc54c626acfd7adf3aac608316a6aa"}
Jul 07 16:23:44.198982 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.198773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fn2kg" event={"ID":"249ad43d-fc4a-4a80-972b-e88b626b20fa","Type":"ContainerStarted","Data":"78ef4b19172f3636f02f4f7564741d6b393588c5f312c390c06edeee3143622e"}
Jul 07 16:23:44.201031 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.200799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prxkt" event={"ID":"a485ff9d-efde-4b4b-949a-27992ce276e0","Type":"ContainerStarted","Data":"32d87014488791d5170e2e6cd3bf8ae1baca703d7bbfb3b8bfd62dc31d5831ac"}
Jul 07 16:23:44.203505 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.203446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tw2zg" event={"ID":"18b23933-59e6-4828-9915-724175f870ca","Type":"ContainerStarted","Data":"e604c7d34dc22f4d728206a40e732ac946cdebf70ae407e788ce67cb6c8eb73f"}
Jul 07 16:23:44.204970 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.204948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwspc" event={"ID":"2f711f0e-5198-4415-a6fe-63ff5cfb9b1d","Type":"ContainerStarted","Data":"7274790e3cf5a062db8cbf071c9b0887b82e2ee0698661c1959df001df581388"}
Jul 07 16:23:44.206431 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.206367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c62q6" event={"ID":"46480d98-8a3a-4a69-8214-db6d53f2f9d3","Type":"ContainerStarted","Data":"0e62b4b0b2405e19c1c03233ba3eae5e700f64adba2b8925d63432b618b92970"}
Jul 07 16:23:44.208637 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.208612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"afbe933bc2227177a0cab3b666b8ac7eef02e8dac77558a5a1eab8a3f982734e"}
Jul 07 16:23:44.208714 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.208687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"9917d4186df133ba7b788f248adafcd694412563026b6e34ea108ad930183b25"}
Jul 07 16:23:44.219339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.219304    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" podStartSLOduration=22.219293693 podStartE2EDuration="22.219293693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:27.177125827 +0000 UTC m=+5.710042475" watchObservedRunningTime="2026-07-07 16:23:44.219293693 +0000 UTC m=+22.752210339"
Jul 07 16:23:44.250559 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.250523    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fn2kg" podStartSLOduration=4.361245387 podStartE2EDuration="23.25050855s" totalImagesPullingTime="18.889263163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.23369775 +0000 UTC m=+22.766614396" watchObservedRunningTime="2026-07-07 16:23:44.25050855 +0000 UTC m=+22.783425190"
Jul 07 16:23:44.250942 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.250917    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tw2zg" podStartSLOduration=9.248320277 podStartE2EDuration="23.250903607s" totalImagesPullingTime="14.00258333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.250619831 +0000 UTC m=+22.783536458" watchObservedRunningTime="2026-07-07 16:23:44.250903607 +0000 UTC m=+22.783820251"
Jul 07 16:23:44.269032 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.268993    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c62q6" podStartSLOduration=4.378033679 podStartE2EDuration="23.268979495s" totalImagesPullingTime="18.890945816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.26599295 +0000 UTC m=+22.798909597" watchObservedRunningTime="2026-07-07 16:23:44.268979495 +0000 UTC m=+22.801896139"
Jul 07 16:23:44.285021 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.284987    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-prxkt" podStartSLOduration=4.392564688 podStartE2EDuration="23.284974468s" totalImagesPullingTime="18.89240978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.284722347 +0000 UTC m=+22.817638992" watchObservedRunningTime="2026-07-07 16:23:44.284974468 +0000 UTC m=+22.817891112"
Jul 07 16:23:44.309322 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:44.309284    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bwspc" podStartSLOduration=4.26931326 podStartE2EDuration="23.309269328s" totalImagesPullingTime="19.039956068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.308997292 +0000 UTC m=+22.841914111" watchObservedRunningTime="2026-07-07 16:23:44.309269328 +0000 UTC m=+22.842185973"
Jul 07 16:23:45.212881 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.212800    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="de93c71f611472e5403f87f84a16bd762785dc5a99ca3f49c0efbbb08cb24705" exitCode=0
Jul 07 16:23:45.213800 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.212924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"de93c71f611472e5403f87f84a16bd762785dc5a99ca3f49c0efbbb08cb24705"}
Jul 07 16:23:45.216619 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.216580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"cd61a1e5ebf3ad5d5f263afa620c9e707a5c8bd81ed15ee905403ea1779ea363"}
Jul 07 16:23:45.216739 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.216671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"b96f67399fe0174cab7f956a581ed7fbbd15c696b73f5f3b3decca5721707c38"}
Jul 07 16:23:45.216739 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.216719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"af88c96ba3344c73d62e8419a810a6a3296768874f169b305a5c2c5a2e517dc0"}
Jul 07 16:23:45.216969 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.216766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"9a62cc1fd56c31acdb533cc402d77649c02b9e54562987964f69210cf58fb174"}
Jul 07 16:23:45.374590 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.374565    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 16:23:45.933608 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.933501    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:23:45.374586837Z","UUID":"d7a96562-b8d0-4192-9d64-7812e0df17d9","Handler":null,"Name":"","Endpoint":""}
Jul 07 16:23:45.935299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.935274    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 16:23:45.935431 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:45.935326    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 16:23:46.089258 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.089189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:46.089419 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:46.089299    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:46.089419 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.089312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:46.089419 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.089190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:46.089565 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:46.089429    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:46.089565 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:46.089525    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:46.187717 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.187657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:46.188433 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.188405    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:46.219863 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.219827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99ln6" event={"ID":"42c85b9b-6749-4c66-bf67-370af90c338a","Type":"ContainerStarted","Data":"6d181d49acad11f992097f7728796305493e5794d517a4af4f0342c7807128c3"}
Jul 07 16:23:46.222406 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.222381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq" event={"ID":"6b60ed0b-dbd8-48c5-9309-bc83f538afe4","Type":"ContainerStarted","Data":"ce4961c5a87e56819585dc6798e5000b4489f6c40a10f34d496a5531f35b662d"}
Jul 07 16:23:46.236136 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:46.236090    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-99ln6" podStartSLOduration=6.349156499 podStartE2EDuration="25.236072301s" totalImagesPullingTime="18.886915802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:46.235360299 +0000 UTC m=+24.768276945" watchObservedRunningTime="2026-07-07 16:23:46.236072301 +0000 UTC m=+24.768988947"
Jul 07 16:23:47.227486 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:47.227329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq" event={"ID":"6b60ed0b-dbd8-48c5-9309-bc83f538afe4","Type":"ContainerStarted","Data":"11cbfa1fd5a86be38a594ab02b08fc5e2b3fd5433a81b0ee3e6aa561d5a07c90"}
Jul 07 16:23:47.231024 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:47.230996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"eae17960a4fee3271c85de85a80928c5238d40c8722827ae1cc1feb67cc0d3b3"}
Jul 07 16:23:47.231150 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:47.231032    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:23:47.269566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:47.269524    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b5zq" podStartSLOduration=5.231641984 podStartE2EDuration="26.269509587s" totalImagesPullingTime="21.037867603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:47.264051703 +0000 UTC m=+25.796968351" watchObservedRunningTime="2026-07-07 16:23:47.269509587 +0000 UTC m=+25.802426232"
Jul 07 16:23:48.090049 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:48.090013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:48.090226 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:48.090022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:48.090226 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:48.090140    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:48.090226 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:48.090028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:48.090353 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:48.090223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:48.090353 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:48.090334    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:49.236453 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.236282    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="7d71fb5be52ebfff148f3789634773b1c344aca43007e08b78e34c4c4fdf89d9" exitCode=0
Jul 07 16:23:49.237137 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.236341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"7d71fb5be52ebfff148f3789634773b1c344aca43007e08b78e34c4c4fdf89d9"}
Jul 07 16:23:49.242901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.242877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" event={"ID":"ac619709-6f42-4267-9cee-72489b234fce","Type":"ContainerStarted","Data":"4af26a707a74430aa48c258178419ae9bb0524ed6579d38814acb37e6a549fb0"}
Jul 07 16:23:49.243156 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.243138    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:49.243256 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.243222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:49.258228 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.258211    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:49.294736 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:49.294689    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z" podStartSLOduration=9.352671446 podStartE2EDuration="28.294675996s" totalImagesPullingTime="18.94200455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:49.2934791 +0000 UTC m=+27.826395746" watchObservedRunningTime="2026-07-07 16:23:49.294675996 +0000 UTC m=+27.827592706"
Jul 07 16:23:50.091727 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.091474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:50.091953 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.091860    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:50.092277 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.092253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:50.092378 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.092351    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:50.092439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.092424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:50.092534 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.092512    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:50.245863 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.245832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:50.261157 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.261134    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:23:50.811945 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.811869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rrzf"]
Jul 07 16:23:50.812100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.811966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:50.812100 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.812054    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:50.812396 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.812378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8h5z"]
Jul 07 16:23:50.812470 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.812459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:50.812551 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.812535    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:50.814009 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.813985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvnp6"]
Jul 07 16:23:50.814136 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:50.814070    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:50.814205 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:50.814163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:51.250137 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:51.250105    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="e23c9e79cb5927bcf965e6765f50fd7398407217945dba6a58ded88d0667edcf" exitCode=0
Jul 07 16:23:51.250529 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:51.250191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"e23c9e79cb5927bcf965e6765f50fd7398407217945dba6a58ded88d0667edcf"}
Jul 07 16:23:52.089972 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:52.089941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:52.090112 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:52.090006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:52.090112 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:52.090054    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:52.090112 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:52.090089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:53.089826 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.089639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:53.090181 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:53.089918    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:53.190535 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.190461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:53.190679 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.190618    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:23:53.191142 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.191116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tw2zg"
Jul 07 16:23:53.257802 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.257750    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="1735ad7371dd1dd8dd27135e58462968879cc395a4da8155930b69e3128b3d41" exitCode=0
Jul 07 16:23:53.257964 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:53.257824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"1735ad7371dd1dd8dd27135e58462968879cc395a4da8155930b69e3128b3d41"}
Jul 07 16:23:54.089955 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:54.089926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:54.090542 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:54.089926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:54.090542 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:54.090051    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:54.090542 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:54.090132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:55.089848 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:55.089822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:55.090014 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.089945    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:23:55.611484 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:55.611457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:55.611626 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:55.611504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:55.611626 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.611599    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:55.611711 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.611648    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret podName:5ee74c68-b023-4f56-8591-8449f020992f nodeName:}" failed. No retries permitted until 2026-07-07 16:24:27.61163495 +0000 UTC m=+66.144551572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret") pod "global-pull-secret-syncer-6rrzf" (UID: "5ee74c68-b023-4f56-8591-8449f020992f") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:23:55.611711 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.611599    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:55.611809 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.611715    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:24:27.61170124 +0000 UTC m=+66.144617878 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:23:55.712600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:55.712568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:55.712757 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.712734    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:23:55.712757 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.712757    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:23:55.712867 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.712771    2565 projected.go:196] Error preparing data for projected volume kube-api-access-69wtg for pod openshift-network-diagnostics/network-check-target-bvnp6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:55.712867 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:55.712835    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg podName:2eb484ed-24ee-4ae4-a340-3975049e98eb nodeName:}" failed. No retries permitted until 2026-07-07 16:24:27.712821405 +0000 UTC m=+66.245738027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-69wtg" (UniqueName: "kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg") pod "network-check-target-bvnp6" (UID: "2eb484ed-24ee-4ae4-a340-3975049e98eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:23:56.089583 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.089554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:56.089778 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.089554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:56.089778 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:56.089682    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvnp6" podUID="2eb484ed-24ee-4ae4-a340-3975049e98eb"
Jul 07 16:23:56.089778 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:56.089726    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rrzf" podUID="5ee74c68-b023-4f56-8591-8449f020992f"
Jul 07 16:23:56.706250 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.706221    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeReady"
Jul 07 16:23:56.706728 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.706340    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 16:23:56.761349 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.761326    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lnwzn"]
Jul 07 16:23:56.764343 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.764319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:56.764481 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.764385    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sp2sb"]
Jul 07 16:23:56.766355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.766335    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:56.767678 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.767662    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 16:23:56.767678 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.767671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 16:23:56.768017 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.768002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f9fh\""
Jul 07 16:23:56.769330 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.769312    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 16:23:56.769671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.769652    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 16:23:56.770078 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.770058    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 16:23:56.770176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.770130    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f8xb4\""
Jul 07 16:23:56.775029 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.775009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnwzn"]
Jul 07 16:23:56.778298 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.778276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sp2sb"]
Jul 07 16:23:56.922276 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lhr\" (UniqueName: \"kubernetes.io/projected/05439776-992c-487e-b154-0cb34885da99-kube-api-access-v8lhr\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:56.922412 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05439776-992c-487e-b154-0cb34885da99-config-volume\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:56.922412 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05439776-992c-487e-b154-0cb34885da99-tmp-dir\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:56.922412 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:56.922539 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09972be-b56f-4b94-9583-6efcaeb8ce09-cert\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:56.922572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:56.922535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28c7b\" (UniqueName: \"kubernetes.io/projected/f09972be-b56f-4b94-9583-6efcaeb8ce09-kube-api-access-28c7b\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.022732 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lhr\" (UniqueName: \"kubernetes.io/projected/05439776-992c-487e-b154-0cb34885da99-kube-api-access-v8lhr\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.022874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05439776-992c-487e-b154-0cb34885da99-config-volume\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.022874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05439776-992c-487e-b154-0cb34885da99-tmp-dir\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.022874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.022874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09972be-b56f-4b94-9583-6efcaeb8ce09-cert\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.023087 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.022885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28c7b\" (UniqueName: \"kubernetes.io/projected/f09972be-b56f-4b94-9583-6efcaeb8ce09-kube-api-access-28c7b\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.023087 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:57.022972    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:23:57.023087 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:57.023054    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:57.523033737 +0000 UTC m=+36.055950366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:23:57.023361 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.023338    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05439776-992c-487e-b154-0cb34885da99-tmp-dir\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.023464 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.023445    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05439776-992c-487e-b154-0cb34885da99-config-volume\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.027179 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.027152    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09972be-b56f-4b94-9583-6efcaeb8ce09-cert\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.035309 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.035244    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lhr\" (UniqueName: \"kubernetes.io/projected/05439776-992c-487e-b154-0cb34885da99-kube-api-access-v8lhr\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.035566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.035543    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28c7b\" (UniqueName: \"kubernetes.io/projected/f09972be-b56f-4b94-9583-6efcaeb8ce09-kube-api-access-28c7b\") pod \"ingress-canary-sp2sb\" (UID: \"f09972be-b56f-4b94-9583-6efcaeb8ce09\") " pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.083542 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.083522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sp2sb"
Jul 07 16:23:57.089591 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.089572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:23:57.092398 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.092377    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:23:57.092481 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.092415    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sncwv\""
Jul 07 16:23:57.248055 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.247867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sp2sb"]
Jul 07 16:23:57.251665 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:23:57.251631    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09972be_b56f_4b94_9583_6efcaeb8ce09.slice/crio-ea15604ddb8e6d7983c2d9887159f8f7cd001ed678c1c8e2719959c8fe76d6a1 WatchSource:0}: Error finding container ea15604ddb8e6d7983c2d9887159f8f7cd001ed678c1c8e2719959c8fe76d6a1: Status 404 returned error can't find the container with id ea15604ddb8e6d7983c2d9887159f8f7cd001ed678c1c8e2719959c8fe76d6a1
Jul 07 16:23:57.269179 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.269152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sp2sb" event={"ID":"f09972be-b56f-4b94-9583-6efcaeb8ce09","Type":"ContainerStarted","Data":"ea15604ddb8e6d7983c2d9887159f8f7cd001ed678c1c8e2719959c8fe76d6a1"}
Jul 07 16:23:57.271954 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.271931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerStarted","Data":"0174b6fc4525eb317ae85f020140c7a24fea2840421a28d09018281d083bb303"}
Jul 07 16:23:57.527425 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:57.527396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:57.527545 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:57.527530    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:23:57.527594 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:57.527588    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:58.527572309 +0000 UTC m=+37.060488946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:23:58.090053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.090019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:23:58.090665 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.090244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:23:58.095712 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.095685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:23:58.095869 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.095685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:23:58.095869 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.095772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkpxx\""
Jul 07 16:23:58.095869 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.095691    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:23:58.277585 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.277556    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="0174b6fc4525eb317ae85f020140c7a24fea2840421a28d09018281d083bb303" exitCode=0
Jul 07 16:23:58.277744 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.277602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"0174b6fc4525eb317ae85f020140c7a24fea2840421a28d09018281d083bb303"}
Jul 07 16:23:58.536529 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:58.536498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:23:58.536672 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:58.536644    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:23:58.536715 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:23:58.536701    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:00.536684156 +0000 UTC m=+39.069600797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:23:59.284562 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:59.284517    2565 generic.go:356] "Generic (PLEG): container finished" podID="50299ae9-ceb8-450e-9854-159da1baf43e" containerID="26995af771458766119b642367147a9ca8883ccffee027b5091b14e24c3a3a82" exitCode=0
Jul 07 16:23:59.285106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:23:59.284577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerDied","Data":"26995af771458766119b642367147a9ca8883ccffee027b5091b14e24c3a3a82"}
Jul 07 16:24:00.289901 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:00.289646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr42w" event={"ID":"50299ae9-ceb8-450e-9854-159da1baf43e","Type":"ContainerStarted","Data":"2e364d0a81ed94e007391f1938e6eb8238de74837858fb212e9fee26ccfaf8b8"}
Jul 07 16:24:00.291111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:00.291083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sp2sb" event={"ID":"f09972be-b56f-4b94-9583-6efcaeb8ce09","Type":"ContainerStarted","Data":"3444dc3fc706589181f940513a47d253abe729c3fda71a96a30debf41995e181"}
Jul 07 16:24:00.314311 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:00.314260    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kr42w" podStartSLOduration=9.596389063 podStartE2EDuration="39.314242928s" totalImagesPullingTime="29.717853865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:00.312394779 +0000 UTC m=+38.845311426" watchObservedRunningTime="2026-07-07 16:24:00.314242928 +0000 UTC m=+38.847159575"
Jul 07 16:24:00.328777 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:00.328721    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sp2sb" podStartSLOduration=1.86643536 podStartE2EDuration="4.328708567s" totalImagesPullingTime="2.462273207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:00.327888271 +0000 UTC m=+38.860804916" watchObservedRunningTime="2026-07-07 16:24:00.328708567 +0000 UTC m=+38.861625206"
Jul 07 16:24:00.548453 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:00.548405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:24:00.548540 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:00.548486    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:24:00.548540 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:00.548531    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:04.548519797 +0000 UTC m=+43.081436420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:24:04.572462 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:04.572423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:24:04.573052 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:04.572545    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:24:04.573052 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:04.572596    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:12.572583594 +0000 UTC m=+51.105500216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:24:12.629935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:12.629903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:24:12.630283 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:12.630021    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:24:12.630283 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:12.630076    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:28.630063155 +0000 UTC m=+67.162979777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:24:22.262628 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:22.262599    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm47z"
Jul 07 16:24:27.633064 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.633028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:24:27.633487 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.633077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:24:27.636241 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.636223    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:24:27.636984 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.636966    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:24:27.643525 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:27.643508    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 16:24:27.643607 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:27.643595    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:25:31.64357325 +0000 UTC m=+130.176489890 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : secret "metrics-daemon-secret" not found
Jul 07 16:24:27.646995 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.646971    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee74c68-b023-4f56-8591-8449f020992f-original-pull-secret\") pod \"global-pull-secret-syncer-6rrzf\" (UID: \"5ee74c68-b023-4f56-8591-8449f020992f\") " pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:24:27.733814 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.733776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:24:27.737108 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.737092    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:24:27.748373 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.748352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:24:27.757520 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.757466    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wtg\" (UniqueName: \"kubernetes.io/projected/2eb484ed-24ee-4ae4-a340-3975049e98eb-kube-api-access-69wtg\") pod \"network-check-target-bvnp6\" (UID: \"2eb484ed-24ee-4ae4-a340-3975049e98eb\") " pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:24:27.806013 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.805988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkpxx\""
Jul 07 16:24:27.807036 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.807022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rrzf"
Jul 07 16:24:27.813812 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.813798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:24:27.968407 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.968372    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rrzf"]
Jul 07 16:24:27.971050 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:24:27.971029    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee74c68_b023_4f56_8591_8449f020992f.slice/crio-c359f68b958f6f82dc1dbaeb3eb70690c246f69d9847a7d01445038574b64cc4 WatchSource:0}: Error finding container c359f68b958f6f82dc1dbaeb3eb70690c246f69d9847a7d01445038574b64cc4: Status 404 returned error can't find the container with id c359f68b958f6f82dc1dbaeb3eb70690c246f69d9847a7d01445038574b64cc4
Jul 07 16:24:27.983391 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:27.983370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvnp6"]
Jul 07 16:24:27.987003 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:24:27.986982    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb484ed_24ee_4ae4_a340_3975049e98eb.slice/crio-65a4b9ef6d647f1ba24c5e22b97c4c2fb237963d9d7a9dc5d3058f1a3390f3d7 WatchSource:0}: Error finding container 65a4b9ef6d647f1ba24c5e22b97c4c2fb237963d9d7a9dc5d3058f1a3390f3d7: Status 404 returned error can't find the container with id 65a4b9ef6d647f1ba24c5e22b97c4c2fb237963d9d7a9dc5d3058f1a3390f3d7
Jul 07 16:24:28.355033 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:28.355008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rrzf" event={"ID":"5ee74c68-b023-4f56-8591-8449f020992f","Type":"ContainerStarted","Data":"c359f68b958f6f82dc1dbaeb3eb70690c246f69d9847a7d01445038574b64cc4"}
Jul 07 16:24:28.356023 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:28.356000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvnp6" event={"ID":"2eb484ed-24ee-4ae4-a340-3975049e98eb","Type":"ContainerStarted","Data":"65a4b9ef6d647f1ba24c5e22b97c4c2fb237963d9d7a9dc5d3058f1a3390f3d7"}
Jul 07 16:24:28.639365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:28.639267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:24:28.639779 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:28.639407    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:24:28.639779 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:24:28.639483    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:25:00.639463591 +0000 UTC m=+99.172380217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:24:33.370943 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:33.370904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvnp6" event={"ID":"2eb484ed-24ee-4ae4-a340-3975049e98eb","Type":"ContainerStarted","Data":"eef4fcb9d24a1550bfc5cdfe3729c8f57da4a775c6baded2a23402911a9e66c7"}
Jul 07 16:24:33.371468 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:33.371015    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:24:33.372409 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:33.372386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rrzf" event={"ID":"5ee74c68-b023-4f56-8591-8449f020992f","Type":"ContainerStarted","Data":"588b71355249454412e718ff0f812c485effeca1bbc3dd4397750c309483ffaf"}
Jul 07 16:24:33.391252 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:33.391211    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bvnp6" podStartSLOduration=67.913616985 podStartE2EDuration="1m12.391198637s" totalImagesPullingTime="4.477581652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:33.390975704 +0000 UTC m=+71.923892348" watchObservedRunningTime="2026-07-07 16:24:33.391198637 +0000 UTC m=+71.924115276"
Jul 07 16:24:33.409174 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:33.409134    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6rrzf" podStartSLOduration=67.912446676 podStartE2EDuration="1m12.409121401s" totalImagesPullingTime="4.496674725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:33.408636279 +0000 UTC m=+71.941552923" watchObservedRunningTime="2026-07-07 16:24:33.409121401 +0000 UTC m=+71.942038045"
Jul 07 16:24:37.778759 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.778728    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"]
Jul 07 16:24:37.780771 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.780755    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"]
Jul 07 16:24:37.780890 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.780873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.782874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.782852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.783537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.783519    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 16:24:37.783629 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.783611    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 16:24:37.783851 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.783838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 16:24:37.786335 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.786310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 16:24:37.786523 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.786507    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 16:24:37.786715 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.786702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 16:24:37.787694 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.787675    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 16:24:37.787925 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.787910    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 16:24:37.798870 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.798851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"]
Jul 07 16:24:37.805148 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558f84d-87a1-48e9-b396-b350dba754c0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.805356 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfp4\" (UniqueName: \"kubernetes.io/projected/7558f84d-87a1-48e9-b396-b350dba754c0-kube-api-access-9sfp4\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.805719 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7txd\" (UniqueName: \"kubernetes.io/projected/53463284-a184-4424-b352-970131fa4dd5-kube-api-access-p7txd\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.805810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.805810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558f84d-87a1-48e9-b396-b350dba754c0-tmp\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.805917 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.805917 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53463284-a184-4424-b352-970131fa4dd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.805917 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.805917 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805863    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"]
Jul 07 16:24:37.805917 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.805871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906226 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906226 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558f84d-87a1-48e9-b396-b350dba754c0-tmp\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53463284-a184-4424-b352-970131fa4dd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558f84d-87a1-48e9-b396-b350dba754c0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfp4\" (UniqueName: \"kubernetes.io/projected/7558f84d-87a1-48e9-b396-b350dba754c0-kube-api-access-9sfp4\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.906463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7txd\" (UniqueName: \"kubernetes.io/projected/53463284-a184-4424-b352-970131fa4dd5-kube-api-access-p7txd\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.906822 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.906732    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558f84d-87a1-48e9-b396-b350dba754c0-tmp\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.907111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.907085    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53463284-a184-4424-b352-970131fa4dd5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.908948 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.908923    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.909047 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.908923    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-ca\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.909089 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.909070    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558f84d-87a1-48e9-b396-b350dba754c0-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.909252 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.909234    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.909387 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.909368    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53463284-a184-4424-b352-970131fa4dd5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:37.916109 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.916078    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfp4\" (UniqueName: \"kubernetes.io/projected/7558f84d-87a1-48e9-b396-b350dba754c0-kube-api-access-9sfp4\") pod \"klusterlet-addon-workmgr-d9c866548-hvfrs\" (UID: \"7558f84d-87a1-48e9-b396-b350dba754c0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:37.916356 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:37.916338    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7txd\" (UniqueName: \"kubernetes.io/projected/53463284-a184-4424-b352-970131fa4dd5-kube-api-access-p7txd\") pod \"cluster-proxy-proxy-agent-5974d5d669-bbvpx\" (UID: \"53463284-a184-4424-b352-970131fa4dd5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:38.090950 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.090909    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:38.106111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.106087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:24:38.240668 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.240639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"]
Jul 07 16:24:38.243536 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:24:38.243510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7558f84d_87a1_48e9_b396_b350dba754c0.slice/crio-d9a958505eb094015acbf5b137596d83373b4f8435d29a1129fd59c595ca9bc5 WatchSource:0}: Error finding container d9a958505eb094015acbf5b137596d83373b4f8435d29a1129fd59c595ca9bc5: Status 404 returned error can't find the container with id d9a958505eb094015acbf5b137596d83373b4f8435d29a1129fd59c595ca9bc5
Jul 07 16:24:38.247124 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.247096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"]
Jul 07 16:24:38.250640 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:24:38.250619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53463284_a184_4424_b352_970131fa4dd5.slice/crio-829c746671c8f8973547154a939d3ce237c26e0756756e183adb00ff7bdc73bb WatchSource:0}: Error finding container 829c746671c8f8973547154a939d3ce237c26e0756756e183adb00ff7bdc73bb: Status 404 returned error can't find the container with id 829c746671c8f8973547154a939d3ce237c26e0756756e183adb00ff7bdc73bb
Jul 07 16:24:38.385267 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.385188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs" event={"ID":"7558f84d-87a1-48e9-b396-b350dba754c0","Type":"ContainerStarted","Data":"d9a958505eb094015acbf5b137596d83373b4f8435d29a1129fd59c595ca9bc5"}
Jul 07 16:24:38.386212 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:38.386183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerStarted","Data":"829c746671c8f8973547154a939d3ce237c26e0756756e183adb00ff7bdc73bb"}
Jul 07 16:24:44.404157 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.404119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs" event={"ID":"7558f84d-87a1-48e9-b396-b350dba754c0","Type":"ContainerStarted","Data":"08698f0d81885c3bfdd5943e9645b8d79a312b9774675ee2f44583103a8fb844"}
Jul 07 16:24:44.404570 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.404179    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:44.405827 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.405807    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:24:44.406344 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.406326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerStarted","Data":"be217dd2535fab499f845597e382aa697c356860d8451943608f9063a7408dc7"}
Jul 07 16:24:44.406413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.406353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerStarted","Data":"9def4c67f7a0d93dda1c5a07feb2e96df00a961a6e66f3b2b1aceb9d230984b5"}
Jul 07 16:24:44.406413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.406368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerStarted","Data":"60bb3e5d4b0daf588aa5282fca87951be1df31dddb8293ddf19f8ce4bd2b56af"}
Jul 07 16:24:44.423223 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.423174    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs" podStartSLOduration=2.299055884 podStartE2EDuration="7.423162575s" totalImagesPullingTime="5.124106691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:44.422534689 +0000 UTC m=+82.955451334" watchObservedRunningTime="2026-07-07 16:24:44.423162575 +0000 UTC m=+82.956079220"
Jul 07 16:24:44.460501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:24:44.460465    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" podStartSLOduration=2.343498 podStartE2EDuration="7.460453972s" totalImagesPullingTime="5.116955972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:44.459672744 +0000 UTC m=+82.992589389" watchObservedRunningTime="2026-07-07 16:24:44.460453972 +0000 UTC m=+82.993370657"
Jul 07 16:25:00.661094 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:25:00.661017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:25:00.661428 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:25:00.661164    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:25:00.661428 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:25:00.661213    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:26:04.66119912 +0000 UTC m=+163.194115748 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:25:04.377172 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:25:04.377143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bvnp6"
Jul 07 16:25:31.664272 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:25:31.664236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:25:31.664832 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:25:31.664404    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 16:25:31.664832 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:25:31.664482    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:27:33.664460787 +0000 UTC m=+252.197377432 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : secret "metrics-daemon-secret" not found
Jul 07 16:25:59.777221 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:25:59.777184    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lnwzn" podUID="05439776-992c-487e-b154-0cb34885da99"
Jul 07 16:26:00.105004 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:26:00.104975    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:26:00.605021 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:00.604997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:26:04.671776 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:04.671737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:26:04.672168 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:26:04.671874    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:26:04.672168 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:26:04.671934    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls podName:05439776-992c-487e-b154-0cb34885da99 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:06.671920905 +0000 UTC m=+285.204837528 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls") pod "dns-default-lnwzn" (UID: "05439776-992c-487e-b154-0cb34885da99") : secret "dns-default-metrics-tls" not found
Jul 07 16:26:15.089468 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:15.089396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:26:43.718439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:43.718383    2565 generic.go:356] "Generic (PLEG): container finished" podID="7558f84d-87a1-48e9-b396-b350dba754c0" containerID="08698f0d81885c3bfdd5943e9645b8d79a312b9774675ee2f44583103a8fb844" exitCode=1
Jul 07 16:26:43.718439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:43.718416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs" event={"ID":"7558f84d-87a1-48e9-b396-b350dba754c0","Type":"ContainerDied","Data":"08698f0d81885c3bfdd5943e9645b8d79a312b9774675ee2f44583103a8fb844"}
Jul 07 16:26:43.718764 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:43.718701    2565 scope.go:117] "RemoveContainer" containerID="08698f0d81885c3bfdd5943e9645b8d79a312b9774675ee2f44583103a8fb844"
Jul 07 16:26:44.404254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:44.404231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:26:44.722968 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:44.722910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs" event={"ID":"7558f84d-87a1-48e9-b396-b350dba754c0","Type":"ContainerStarted","Data":"7ff82f90698964722342b6257c287e04a0e813da31b0628428318177875f0bc6"}
Jul 07 16:26:44.723284 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:44.723104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:26:44.723527 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:26:44.723512    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9c866548-hvfrs"
Jul 07 16:27:33.721870 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:33.721838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:27:33.722262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:33.721945    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 16:27:33.722262 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:33.722001    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs podName:ababdc3c-82b9-467c-91bd-c6464619cb2c nodeName:}" failed. No retries permitted until 2026-07-07 16:29:35.721987276 +0000 UTC m=+374.254903900 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs") pod "network-metrics-daemon-p8h5z" (UID: "ababdc3c-82b9-467c-91bd-c6464619cb2c") : secret "metrics-daemon-secret" not found
Jul 07 16:27:48.257344 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.257271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5sc9g"]
Jul 07 16:27:48.259121 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.259105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.267342 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.267322    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 16:27:48.267450 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.267372    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 16:27:48.267450 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.267378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 16:27:48.268383 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.268365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2w8q7\""
Jul 07 16:27:48.268479 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.268397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 16:27:48.284418 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.284396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sc9g"]
Jul 07 16:27:48.313979 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.313944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-data-volume\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.314111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.313988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-crio-socket\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.314111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.314008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.314111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.314093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68m5\" (UniqueName: \"kubernetes.io/projected/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-api-access-q68m5\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.314208 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.314144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415320 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-data-volume\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415402 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-crio-socket\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415402 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415402 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68m5\" (UniqueName: \"kubernetes.io/projected/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-api-access-q68m5\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415512 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415512 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415437    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-crio-socket\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.415512 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.415478    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:48.415632 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.415537    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:48.915520002 +0000 UTC m=+267.448436629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:48.415632 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.415556    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:27:48.415632 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.415607    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:48.915594608 +0000 UTC m=+267.448511261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : configmap "kube-rbac-proxy" not found
Jul 07 16:27:48.415807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.415626    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-data-volume\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.431212 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.431192    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68m5\" (UniqueName: \"kubernetes.io/projected/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-api-access-q68m5\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.919547 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.919518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.919724 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:48.919561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:48.919724 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.919657    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:27:48.919724 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.919666    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:48.919724 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.919718    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:49.919703309 +0000 UTC m=+268.452619933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : configmap "kube-rbac-proxy" not found
Jul 07 16:27:48.919886 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:48.919732    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:49.91972609 +0000 UTC m=+268.452642712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:49.928021 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:49.927986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:49.928382 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:49.928040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:49.928382 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:49.928134    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:49.928382 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:49.928195    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:51.928181305 +0000 UTC m=+270.461097931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:49.928382 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:49.928134    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:27:49.928382 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:49.928261    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:51.92824997 +0000 UTC m=+270.461166592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : configmap "kube-rbac-proxy" not found
Jul 07 16:27:50.786403 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:50.786375    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fn2kg_249ad43d-fc4a-4a80-972b-e88b626b20fa/dns-node-resolver/0.log"
Jul 07 16:27:51.788605 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:51.788579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c62q6_46480d98-8a3a-4a69-8214-db6d53f2f9d3/node-ca/0.log"
Jul 07 16:27:51.941040 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:51.941006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:51.941210 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:51.941053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:51.941210 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:51.941109    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:27:51.941210 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:51.941140    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:51.941210 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:51.941153    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:55.941139568 +0000 UTC m=+274.474056190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : configmap "kube-rbac-proxy" not found
Jul 07 16:27:51.941210 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:51.941193    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:55.941179342 +0000 UTC m=+274.474095964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:52.786403 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:52.786373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sp2sb_f09972be-b56f-4b94-9583-6efcaeb8ce09/serve-healthcheck-canary/0.log"
Jul 07 16:27:55.966680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:55.966644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:55.967447 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:27:55.966703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:27:55.967447 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:55.966830    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:55.967447 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:55.966906    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:03.966888094 +0000 UTC m=+282.499804721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:27:55.967447 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:55.966830    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:27:55.967447 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:27:55.966954    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:03.966943157 +0000 UTC m=+282.499859783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : configmap "kube-rbac-proxy" not found
Jul 07 16:28:03.606403 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:28:03.606354    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lnwzn" podUID="05439776-992c-487e-b154-0cb34885da99"
Jul 07 16:28:03.934130 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:03.934052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:04.021364 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:04.021332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:04.021478 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:04.021395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:04.021523 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:28:04.021507    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:28:04.021615 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:28:04.021599    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls podName:f22ab0f8-0c44-4106-afa3-13619b6b0dc9 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:20.021574082 +0000 UTC m=+298.554490705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sc9g" (UID: "f22ab0f8-0c44-4106-afa3-13619b6b0dc9") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:28:04.021987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:04.021964    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:06.743172 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:06.743130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:06.745726 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:06.745689    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05439776-992c-487e-b154-0cb34885da99-metrics-tls\") pod \"dns-default-lnwzn\" (UID: \"05439776-992c-487e-b154-0cb34885da99\") " pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:06.938948 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:06.938922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f9fh\""
Jul 07 16:28:06.944667 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:06.944639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:07.079821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:07.079797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnwzn"]
Jul 07 16:28:07.082885 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:07.082865    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05439776_992c_487e_b154_0cb34885da99.slice/crio-54853e0feeb612047eb4118a6dd33ea73e3cf388339900821502c46d1b9d4cac WatchSource:0}: Error finding container 54853e0feeb612047eb4118a6dd33ea73e3cf388339900821502c46d1b9d4cac: Status 404 returned error can't find the container with id 54853e0feeb612047eb4118a6dd33ea73e3cf388339900821502c46d1b9d4cac
Jul 07 16:28:07.946776 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:07.946738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnwzn" event={"ID":"05439776-992c-487e-b154-0cb34885da99","Type":"ContainerStarted","Data":"54853e0feeb612047eb4118a6dd33ea73e3cf388339900821502c46d1b9d4cac"}
Jul 07 16:28:09.956163 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:09.956128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnwzn" event={"ID":"05439776-992c-487e-b154-0cb34885da99","Type":"ContainerStarted","Data":"5bce01b7b54385753456121a4419d826cd6a592ee56e4e320ce92fc555ac7cd6"}
Jul 07 16:28:09.956537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:09.956183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnwzn" event={"ID":"05439776-992c-487e-b154-0cb34885da99","Type":"ContainerStarted","Data":"6466fa8150f70c43c64b1658e00f0150f50ec46d629e2b75f3bf421d2bd461cf"}
Jul 07 16:28:09.956537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:09.956254    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:09.980019 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:09.979971    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lnwzn" podStartSLOduration=252.200018353 podStartE2EDuration="4m13.979956286s" totalImagesPullingTime="1.779937933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:09.979128028 +0000 UTC m=+288.512044674" watchObservedRunningTime="2026-07-07 16:28:09.979956286 +0000 UTC m=+288.512872932"
Jul 07 16:28:18.090493 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:28:18.090448    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p8h5z" podUID="ababdc3c-82b9-467c-91bd-c6464619cb2c"
Jul 07 16:28:19.961725 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:19.961685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lnwzn"
Jul 07 16:28:20.032020 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:20.031983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:20.034320 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:20.034295    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f22ab0f8-0c44-4106-afa3-13619b6b0dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sc9g\" (UID: \"f22ab0f8-0c44-4106-afa3-13619b6b0dc9\") " pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:20.067399 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:20.067377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sc9g"
Jul 07 16:28:20.194610 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:20.194581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sc9g"]
Jul 07 16:28:20.197890 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:20.197865    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22ab0f8_0c44_4106_afa3_13619b6b0dc9.slice/crio-4aea0fb0008eb581aab6c7fd08ac02a243525473b6b711d6867a6fcd7205adb6 WatchSource:0}: Error finding container 4aea0fb0008eb581aab6c7fd08ac02a243525473b6b711d6867a6fcd7205adb6: Status 404 returned error can't find the container with id 4aea0fb0008eb581aab6c7fd08ac02a243525473b6b711d6867a6fcd7205adb6
Jul 07 16:28:20.992549 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:20.992449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sc9g" event={"ID":"f22ab0f8-0c44-4106-afa3-13619b6b0dc9","Type":"ContainerStarted","Data":"4aea0fb0008eb581aab6c7fd08ac02a243525473b6b711d6867a6fcd7205adb6"}
Jul 07 16:28:22.841626 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.841594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:28:22.862324 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.862298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:28:22.862469 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.862392    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.866417 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.866391    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 16:28:22.866594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.866390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 16:28:22.866712 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.866689    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 16:28:22.868195 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.868164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 16:28:22.868195 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.868170    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 16:28:22.868486 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.868432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fg2nq\""
Jul 07 16:28:22.868658 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.868638    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 16:28:22.869278 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.869181    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 16:28:22.955328 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.955434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.955434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.955434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.955434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj7v\" (UniqueName: \"kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:22.955580 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:22.955459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.056654 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.056841 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.056841 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.056841 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.056841 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056806    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.057068 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.056914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj7v\" (UniqueName: \"kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.057470 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.057408    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.057470 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.057459    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.057642 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.057478    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.059560 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.059541    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.059712 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.059684    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.072742 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.072718    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj7v\" (UniqueName: \"kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v\") pod \"console-6d77b8946-827sn\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") " pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:23.173615 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:23.173584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:24.017337 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:24.017315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:28:24.018712 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:24.018697    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:28:24.019142 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:24.019114    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fe313b_f2d0_4f56_a347_dfd840cd9e0a.slice/crio-efbe305adb5a5f6689067e9bd589c220ceaf89373259f98f66962b779c722edf WatchSource:0}: Error finding container efbe305adb5a5f6689067e9bd589c220ceaf89373259f98f66962b779c722edf: Status 404 returned error can't find the container with id efbe305adb5a5f6689067e9bd589c220ceaf89373259f98f66962b779c722edf
Jul 07 16:28:25.007213 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:25.007129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sc9g" event={"ID":"f22ab0f8-0c44-4106-afa3-13619b6b0dc9","Type":"ContainerStarted","Data":"afac0b78c396bc6e8b8299447ec0ee4ab8476f2ef43372f4d2d4428c80a58c5e"}
Jul 07 16:28:25.007213 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:25.007203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sc9g" event={"ID":"f22ab0f8-0c44-4106-afa3-13619b6b0dc9","Type":"ContainerStarted","Data":"864d44d38f73c63556aff8dc7d4cddf9564b0cbd8cee88b6c3870a2f5fe4c786"}
Jul 07 16:28:25.008421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:25.008391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d77b8946-827sn" event={"ID":"66fe313b-f2d0-4f56-a347-dfd840cd9e0a","Type":"ContainerStarted","Data":"efbe305adb5a5f6689067e9bd589c220ceaf89373259f98f66962b779c722edf"}
Jul 07 16:28:27.572068 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.571496    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x5jkq"]
Jul 07 16:28:27.578043 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.578022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.586632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.586280    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 16:28:27.586632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.586522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 16:28:27.586632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.586535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 16:28:27.586632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.586571    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 16:28:27.587390 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.587371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 16:28:27.587502 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.587376    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 16:28:27.590569 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.590553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jxggr\""
Jul 07 16:28:27.692222 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-root\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692357 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-sys\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692357 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-textfile\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692357 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64vh\" (UniqueName: \"kubernetes.io/projected/1cad0191-16ea-47bd-91a7-cdfe66919fb2-kube-api-access-d64vh\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-tls\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-wtmp\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.692648 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.692503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-metrics-client-ca\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-tls\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-wtmp\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-metrics-client-ca\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-root\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793636 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-sys\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793636 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-textfile\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793636 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64vh\" (UniqueName: \"kubernetes.io/projected/1cad0191-16ea-47bd-91a7-cdfe66919fb2-kube-api-access-d64vh\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793872 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793634    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-root\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793872 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793846    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-sys\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.793872 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.793849    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-wtmp\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.794168 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.794128    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-textfile\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.794292 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.794175    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-metrics-client-ca\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.794838 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.794802    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.796023 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.795983    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.796201 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.796183    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cad0191-16ea-47bd-91a7-cdfe66919fb2-node-exporter-tls\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.818991 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.818968    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64vh\" (UniqueName: \"kubernetes.io/projected/1cad0191-16ea-47bd-91a7-cdfe66919fb2-kube-api-access-d64vh\") pod \"node-exporter-x5jkq\" (UID: \"1cad0191-16ea-47bd-91a7-cdfe66919fb2\") " pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:27.890046 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:27.889989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5jkq"
Jul 07 16:28:28.310717 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:28.310695    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cad0191_16ea_47bd_91a7_cdfe66919fb2.slice/crio-d4aa2afa61e6d4822ec4ea85931c5d55f1d28aea3fab5c1986a9c7095b2da2e7 WatchSource:0}: Error finding container d4aa2afa61e6d4822ec4ea85931c5d55f1d28aea3fab5c1986a9c7095b2da2e7: Status 404 returned error can't find the container with id d4aa2afa61e6d4822ec4ea85931c5d55f1d28aea3fab5c1986a9c7095b2da2e7
Jul 07 16:28:29.027121 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:29.027060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sc9g" event={"ID":"f22ab0f8-0c44-4106-afa3-13619b6b0dc9","Type":"ContainerStarted","Data":"d55c645aefbe118f0035a9155baa918310b9552fde56fa8152447fd0858a997a"}
Jul 07 16:28:29.028992 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:29.028954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d77b8946-827sn" event={"ID":"66fe313b-f2d0-4f56-a347-dfd840cd9e0a","Type":"ContainerStarted","Data":"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"}
Jul 07 16:28:29.030243 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:29.030208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5jkq" event={"ID":"1cad0191-16ea-47bd-91a7-cdfe66919fb2","Type":"ContainerStarted","Data":"d4aa2afa61e6d4822ec4ea85931c5d55f1d28aea3fab5c1986a9c7095b2da2e7"}
Jul 07 16:28:29.062952 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:29.062908    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5sc9g" podStartSLOduration=33.086434807 podStartE2EDuration="41.062889466s" totalImagesPullingTime="7.976454659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:29.062648095 +0000 UTC m=+307.595564753" watchObservedRunningTime="2026-07-07 16:28:29.062889466 +0000 UTC m=+307.595806113"
Jul 07 16:28:29.130479 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:29.130413    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d77b8946-827sn" podStartSLOduration=2.841639641 podStartE2EDuration="7.130394074s" totalImagesPullingTime="4.288754433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:29.129690468 +0000 UTC m=+307.662607115" watchObservedRunningTime="2026-07-07 16:28:29.130394074 +0000 UTC m=+307.663310721"
Jul 07 16:28:30.035463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.035430    2565 generic.go:356] "Generic (PLEG): container finished" podID="1cad0191-16ea-47bd-91a7-cdfe66919fb2" containerID="6a214cde514ad6dc6bb9759bd9d85181d8973f5fb99575fd3427ba987ee9ae34" exitCode=0
Jul 07 16:28:30.035893 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.035525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5jkq" event={"ID":"1cad0191-16ea-47bd-91a7-cdfe66919fb2","Type":"ContainerDied","Data":"6a214cde514ad6dc6bb9759bd9d85181d8973f5fb99575fd3427ba987ee9ae34"}
Jul 07 16:28:30.749557 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.749523    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57c6977688-rbtjb"]
Jul 07 16:28:30.753007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.752986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.759010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.758978    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 16:28:30.759440 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.759361    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 16:28:30.759707 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.759681    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 16:28:30.759838 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.759776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k6r7n\""
Jul 07 16:28:30.759894 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.759883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-33ja10ebsb8do\""
Jul 07 16:28:30.759932 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.759899    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 16:28:30.761022 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.761005    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 16:28:30.776524 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.776502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c6977688-rbtjb"]
Jul 07 16:28:30.917926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.917899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.917936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.917958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-grpc-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.918038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.918066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.918092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.918109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5pgd\" (UniqueName: \"kubernetes.io/projected/36593203-87af-48c7-a09b-b68513d1ab10-kube-api-access-d5pgd\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:30.918235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:30.918145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36593203-87af-48c7-a09b-b68513d1ab10-metrics-client-ca\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019124 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019124 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019124 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-grpc-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pgd\" (UniqueName: \"kubernetes.io/projected/36593203-87af-48c7-a09b-b68513d1ab10-kube-api-access-d5pgd\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.019333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.019269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36593203-87af-48c7-a09b-b68513d1ab10-metrics-client-ca\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.020133 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.020080    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36593203-87af-48c7-a09b-b68513d1ab10-metrics-client-ca\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.021914 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.021890    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.022025 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.021975    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-grpc-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.022089 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.022071    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-tls\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.022188 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.022168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.022247 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.022234    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.022311 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.022289    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/36593203-87af-48c7-a09b-b68513d1ab10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.038317 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.038299    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pgd\" (UniqueName: \"kubernetes.io/projected/36593203-87af-48c7-a09b-b68513d1ab10-kube-api-access-d5pgd\") pod \"thanos-querier-57c6977688-rbtjb\" (UID: \"36593203-87af-48c7-a09b-b68513d1ab10\") " pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.041431 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.041406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5jkq" event={"ID":"1cad0191-16ea-47bd-91a7-cdfe66919fb2","Type":"ContainerStarted","Data":"1c801dab753e1b1ac9b06a809fd53a485ec4edfed39d9ea03e9efbea4dc95594"}
Jul 07 16:28:31.041531 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.041462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5jkq" event={"ID":"1cad0191-16ea-47bd-91a7-cdfe66919fb2","Type":"ContainerStarted","Data":"107743df232a35083f687fbca3a0a71f3f11ed9d353027b84bf5773e029798cc"}
Jul 07 16:28:31.062532 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.062510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:31.075585 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.075544    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x5jkq" podStartSLOduration=3.089042719 podStartE2EDuration="4.07553193s" totalImagesPullingTime="923.674335ms" totalInitContainerRuntime="62.814876ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:31.074643599 +0000 UTC m=+309.607560245" watchObservedRunningTime="2026-07-07 16:28:31.07553193 +0000 UTC m=+309.608448575"
Jul 07 16:28:31.227212 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:31.227184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c6977688-rbtjb"]
Jul 07 16:28:31.232646 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:31.232620    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36593203_87af_48c7_a09b_b68513d1ab10.slice/crio-adce20ced9525275d80e577f4768c0c413118312cd1b9172b02d4eeb4e3639af WatchSource:0}: Error finding container adce20ced9525275d80e577f4768c0c413118312cd1b9172b02d4eeb4e3639af: Status 404 returned error can't find the container with id adce20ced9525275d80e577f4768c0c413118312cd1b9172b02d4eeb4e3639af
Jul 07 16:28:32.045087 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.045056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"adce20ced9525275d80e577f4768c0c413118312cd1b9172b02d4eeb4e3639af"}
Jul 07 16:28:32.278802 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.278765    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d4cd94785-5gj62"]
Jul 07 16:28:32.281546 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.281531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.293855 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.293837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 16:28:32.295330 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.295272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 16:28:32.295330 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.295280    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5ql0vkksergrl\""
Jul 07 16:28:32.295330 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.295320    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 16:28:32.295558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.295354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lw9rr\""
Jul 07 16:28:32.295558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.295359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 16:28:32.302062 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.302037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4cd94785-5gj62"]
Jul 07 16:28:32.431928 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.431893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-metrics-server-audit-profiles\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.431943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-tls\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.432045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-client-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.432098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.432137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-audit-log\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.432159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2nf\" (UniqueName: \"kubernetes.io/projected/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-kube-api-access-pn2nf\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.432191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.432177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533233 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-client-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533407 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533407 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-audit-log\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533407 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2nf\" (UniqueName: \"kubernetes.io/projected/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-kube-api-access-pn2nf\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-metrics-server-audit-profiles\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-tls\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533715 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533568    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-audit-log\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.533991 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.533965    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.534289 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.534271    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-metrics-server-audit-profiles\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.535778 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.535762    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-client-ca-bundle\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.536353 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.536332    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-tls\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.536401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.536332    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.545061 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.545038    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2nf\" (UniqueName: \"kubernetes.io/projected/672ae99d-6d82-4b98-8af7-0f8d842d6bf3-kube-api-access-pn2nf\") pod \"metrics-server-5d4cd94785-5gj62\" (UID: \"672ae99d-6d82-4b98-8af7-0f8d842d6bf3\") " pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.589748 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.589731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:32.738290 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:32.738258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4cd94785-5gj62"]
Jul 07 16:28:32.741854 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:32.741812    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672ae99d_6d82_4b98_8af7_0f8d842d6bf3.slice/crio-100e43a5773228e6e9e87f9ac323447d56179cbf111e574718278d1c7a812fb2 WatchSource:0}: Error finding container 100e43a5773228e6e9e87f9ac323447d56179cbf111e574718278d1c7a812fb2: Status 404 returned error can't find the container with id 100e43a5773228e6e9e87f9ac323447d56179cbf111e574718278d1c7a812fb2
Jul 07 16:28:33.049873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:33.049780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62" event={"ID":"672ae99d-6d82-4b98-8af7-0f8d842d6bf3","Type":"ContainerStarted","Data":"100e43a5773228e6e9e87f9ac323447d56179cbf111e574718278d1c7a812fb2"}
Jul 07 16:28:33.089201 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:33.089168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:28:33.174347 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:33.174314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:33.174487 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:33.174383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:33.179179 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:33.179155    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:34.058795 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.058757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:28:34.493873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.493839    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:28:34.497574 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.497549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.509292 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.509267    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 16:28:34.520728 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.520705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:28:34.550254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pbm\" (UniqueName: \"kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.550612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.550452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.650990 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.650949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.650990 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.650990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651224 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651224 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651224 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651378 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651378 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pbm\" (UniqueName: \"kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651770 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651748    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651748    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.651889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.651749    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.652643 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.652620    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.653372 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.653345    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.653673 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.653658    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.671760 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.671737    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pbm\" (UniqueName: \"kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm\") pod \"console-6c7679fddc-vlg2f\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") " pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.809068 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.808979    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:34.967436 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:34.967410    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:28:34.970381 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:28:34.970350    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ea75e6_138e_475e_9306_bce919a44298.slice/crio-54412cb02163d9707308578c3d992fc9ac55ed2d0f5ba98b5388f6d90bd63c6e WatchSource:0}: Error finding container 54412cb02163d9707308578c3d992fc9ac55ed2d0f5ba98b5388f6d90bd63c6e: Status 404 returned error can't find the container with id 54412cb02163d9707308578c3d992fc9ac55ed2d0f5ba98b5388f6d90bd63c6e
Jul 07 16:28:35.060447 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.060344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"322d9ef9c19b96b42e6b1406bdefc8fe1815d2ffaa635029530c4bd5ecb0f427"}
Jul 07 16:28:35.060894 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.060481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"2564ad547e735294c99675b0682b0188d39375ab70ebf079557366fffa672f33"}
Jul 07 16:28:35.060894 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.060526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"f89893af48185094937c8e4f7091a202a24e935cc885f8100e302426d3f42025"}
Jul 07 16:28:35.064944 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.064910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7679fddc-vlg2f" event={"ID":"79ea75e6-138e-475e-9306-bce919a44298","Type":"ContainerStarted","Data":"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"}
Jul 07 16:28:35.065067 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.064999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7679fddc-vlg2f" event={"ID":"79ea75e6-138e-475e-9306-bce919a44298","Type":"ContainerStarted","Data":"54412cb02163d9707308578c3d992fc9ac55ed2d0f5ba98b5388f6d90bd63c6e"}
Jul 07 16:28:35.067496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.067470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62" event={"ID":"672ae99d-6d82-4b98-8af7-0f8d842d6bf3","Type":"ContainerStarted","Data":"587e3bef47daa2abe80d3c19ced5d4e42b12f7801e787f24500260df42a6ff8d"}
Jul 07 16:28:35.094230 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.094148    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c7679fddc-vlg2f" podStartSLOduration=1.094131241 podStartE2EDuration="1.094131241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:35.0923658 +0000 UTC m=+313.625282447" watchObservedRunningTime="2026-07-07 16:28:35.094131241 +0000 UTC m=+313.627047864"
Jul 07 16:28:35.116586 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:35.116528    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62" podStartSLOduration=1.1743804039999999 podStartE2EDuration="3.116513694s" totalImagesPullingTime="1.94213329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:35.116025547 +0000 UTC m=+313.648942193" watchObservedRunningTime="2026-07-07 16:28:35.116513694 +0000 UTC m=+313.649430317"
Jul 07 16:28:36.074015 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:36.073979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"d5e92076ab4bcb018e96ea61a281c2ba36284820f71d406b4e6c76f6ed217b5a"}
Jul 07 16:28:36.074434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:36.074044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"69b39f0a6153c4be0019fd18923a3075808c9031c36edbe537b27b6444871050"}
Jul 07 16:28:36.074434 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:36.074071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" event={"ID":"36593203-87af-48c7-a09b-b68513d1ab10","Type":"ContainerStarted","Data":"6d9e8459980b51fe5bda5328292399fc664e1c390b5de022c32360080d5e4dbd"}
Jul 07 16:28:36.101121 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:36.101062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb" podStartSLOduration=2.333654491 podStartE2EDuration="6.101046848s" totalImagesPullingTime="3.767392357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:36.099433414 +0000 UTC m=+314.632350080" watchObservedRunningTime="2026-07-07 16:28:36.101046848 +0000 UTC m=+314.633963493"
Jul 07 16:28:37.078168 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:37.078139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:43.093958 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:43.093931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57c6977688-rbtjb"
Jul 07 16:28:44.809671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:44.809630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:44.810080 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:44.809708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:44.814056 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:44.814040    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:45.114419 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:45.114392    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:28:45.167839 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:45.167807    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:28:48.107315 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:48.107273    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" podUID="53463284-a184-4424-b352-970131fa4dd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 16:28:52.590696 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:52.590663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:52.591087 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:52.590729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:28:58.107106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:28:58.107073    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" podUID="53463284-a184-4424-b352-970131fa4dd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 16:29:08.107159 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:08.107125    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" podUID="53463284-a184-4424-b352-970131fa4dd5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 16:29:08.107595 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:08.107185    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx"
Jul 07 16:29:08.107706 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:08.107679    2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"be217dd2535fab499f845597e382aa697c356860d8451943608f9063a7408dc7"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 07 16:29:08.107748 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:08.107730    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" podUID="53463284-a184-4424-b352-970131fa4dd5" containerName="service-proxy" containerID="cri-o://be217dd2535fab499f845597e382aa697c356860d8451943608f9063a7408dc7" gracePeriod=30
Jul 07 16:29:09.191380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:09.191336    2565 generic.go:356] "Generic (PLEG): container finished" podID="53463284-a184-4424-b352-970131fa4dd5" containerID="be217dd2535fab499f845597e382aa697c356860d8451943608f9063a7408dc7" exitCode=2
Jul 07 16:29:09.191776 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:09.191405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerDied","Data":"be217dd2535fab499f845597e382aa697c356860d8451943608f9063a7408dc7"}
Jul 07 16:29:09.191776 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:09.191449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5974d5d669-bbvpx" event={"ID":"53463284-a184-4424-b352-970131fa4dd5","Type":"ContainerStarted","Data":"229c36eff2e0602f197705b738fd9d586f42bf1e6fc4dfadd45b7a4bb2a6ef7f"}
Jul 07 16:29:10.188912 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.188863    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d77b8946-827sn" podUID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" containerName="console" containerID="cri-o://1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d" gracePeriod=15
Jul 07 16:29:10.452937 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.451740    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:29:10.532161 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532217    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmj7v\" (UniqueName: \"kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532338    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config\") pod \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\" (UID: \"66fe313b-f2d0-4f56-a347-dfd840cd9e0a\") "
Jul 07 16:29:10.532558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532480    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:29:10.532663 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532589    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:10.532663 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532586    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca" (OuterVolumeSpecName: "service-ca") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:29:10.532663 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.532628    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config" (OuterVolumeSpecName: "console-config") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:29:10.534684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.534656    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:29:10.534684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.534670    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:29:10.534857 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.534711    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v" (OuterVolumeSpecName: "kube-api-access-tmj7v") pod "66fe313b-f2d0-4f56-a347-dfd840cd9e0a" (UID: "66fe313b-f2d0-4f56-a347-dfd840cd9e0a"). InnerVolumeSpecName "kube-api-access-tmj7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:29:10.633413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.633301    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:10.633413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.633337    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:10.633413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.633353    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:10.633413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.633370    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmj7v\" (UniqueName: \"kubernetes.io/projected/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-kube-api-access-tmj7v\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:10.633413 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:10.633384    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66fe313b-f2d0-4f56-a347-dfd840cd9e0a-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:29:11.202558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.202524    2565 generic.go:356] "Generic (PLEG): container finished" podID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" containerID="1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d" exitCode=0
Jul 07 16:29:11.202730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.202588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d77b8946-827sn"
Jul 07 16:29:11.202730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.202612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d77b8946-827sn" event={"ID":"66fe313b-f2d0-4f56-a347-dfd840cd9e0a","Type":"ContainerDied","Data":"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"}
Jul 07 16:29:11.202730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.202660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d77b8946-827sn" event={"ID":"66fe313b-f2d0-4f56-a347-dfd840cd9e0a","Type":"ContainerDied","Data":"efbe305adb5a5f6689067e9bd589c220ceaf89373259f98f66962b779c722edf"}
Jul 07 16:29:11.202730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.202680    2565 scope.go:117] "RemoveContainer" containerID="1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"
Jul 07 16:29:11.212952 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.212925    2565 scope.go:117] "RemoveContainer" containerID="1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"
Jul 07 16:29:11.213526 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:29:11.213493    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d\": container with ID starting with 1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d not found: ID does not exist" containerID="1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"
Jul 07 16:29:11.213601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.213538    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d"} err="failed to get container status \"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d\": rpc error: code = NotFound desc = could not find container \"1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d\": container with ID starting with 1c85d6dcba36b33d55bfdd9b8fe4367c3e65067a53fe88ccd4c833388e6c4e0d not found: ID does not exist"
Jul 07 16:29:11.230131 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.230099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:29:11.234005 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:11.233983    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d77b8946-827sn"]
Jul 07 16:29:12.093622 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:12.093587    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" path="/var/lib/kubelet/pods/66fe313b-f2d0-4f56-a347-dfd840cd9e0a/volumes"
Jul 07 16:29:12.595171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:12.595144    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:29:12.598739 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:12.598717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d4cd94785-5gj62"
Jul 07 16:29:35.822301 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:35.822260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:29:35.824453 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:35.824432    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ababdc3c-82b9-467c-91bd-c6464619cb2c-metrics-certs\") pod \"network-metrics-daemon-p8h5z\" (UID: \"ababdc3c-82b9-467c-91bd-c6464619cb2c\") " pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:29:36.093393 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:36.093310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sncwv\""
Jul 07 16:29:36.100722 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:36.100705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8h5z"
Jul 07 16:29:36.224823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:36.224775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8h5z"]
Jul 07 16:29:36.227639 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:29:36.227613    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podababdc3c_82b9_467c_91bd_c6464619cb2c.slice/crio-f120a3128ad55617d8b1af9a6dfd52a7f544c6afd53e1836e1e67a06b3abc7be WatchSource:0}: Error finding container f120a3128ad55617d8b1af9a6dfd52a7f544c6afd53e1836e1e67a06b3abc7be: Status 404 returned error can't find the container with id f120a3128ad55617d8b1af9a6dfd52a7f544c6afd53e1836e1e67a06b3abc7be
Jul 07 16:29:36.289581 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:36.289548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8h5z" event={"ID":"ababdc3c-82b9-467c-91bd-c6464619cb2c","Type":"ContainerStarted","Data":"f120a3128ad55617d8b1af9a6dfd52a7f544c6afd53e1836e1e67a06b3abc7be"}
Jul 07 16:29:38.300174 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:38.300140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8h5z" event={"ID":"ababdc3c-82b9-467c-91bd-c6464619cb2c","Type":"ContainerStarted","Data":"4df9462fcba25bc7591493e31d963112a1775db60d4a8b259d856f88f33b76d1"}
Jul 07 16:29:38.300537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:38.300197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8h5z" event={"ID":"ababdc3c-82b9-467c-91bd-c6464619cb2c","Type":"ContainerStarted","Data":"d588ddfddd203c14639a0d31b41a0061051b62b3d1f03b82b1e6a7ba26affd7b"}
Jul 07 16:29:38.324509 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:29:38.324431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8h5z" podStartSLOduration=376.340638223 podStartE2EDuration="6m17.324415503s" totalImagesPullingTime="983.77728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:29:38.322798826 +0000 UTC m=+376.855715482" watchObservedRunningTime="2026-07-07 16:29:38.324415503 +0000 UTC m=+376.857332175"
Jul 07 16:30:07.775395 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:07.775358    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:30:32.795250 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:32.795188    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c7679fddc-vlg2f" podUID="79ea75e6-138e-475e-9306-bce919a44298" containerName="console" containerID="cri-o://4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf" gracePeriod=15
Jul 07 16:30:33.031816 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.031764    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:30:33.117240 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4pbm\" (UniqueName: \"kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117414 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117414 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117414 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117636 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config\") pod \"79ea75e6-138e-475e-9306-bce919a44298\" (UID: \"79ea75e6-138e-475e-9306-bce919a44298\") "
Jul 07 16:30:33.117751 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117726    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca" (OuterVolumeSpecName: "service-ca") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:30:33.117839 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117766    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:30:33.117839 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117811    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:30:33.117930 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117853    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.117930 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117867    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.117930 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.117889    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config" (OuterVolumeSpecName: "console-config") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:30:33.119530 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.119497    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:30:33.119626 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.119502    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm" (OuterVolumeSpecName: "kube-api-access-f4pbm") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "kube-api-access-f4pbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:30:33.119626 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.119561    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79ea75e6-138e-475e-9306-bce919a44298" (UID: "79ea75e6-138e-475e-9306-bce919a44298"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:30:33.218365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.218312    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4pbm\" (UniqueName: \"kubernetes.io/projected/79ea75e6-138e-475e-9306-bce919a44298-kube-api-access-f4pbm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.218365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.218356    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-trusted-ca-bundle\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.218365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.218367    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.218365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.218375    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ea75e6-138e-475e-9306-bce919a44298-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.218365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.218384    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ea75e6-138e-475e-9306-bce919a44298-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:30:33.491556 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.491467    2565 generic.go:356] "Generic (PLEG): container finished" podID="79ea75e6-138e-475e-9306-bce919a44298" containerID="4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf" exitCode=0
Jul 07 16:30:33.491556 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.491510    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7679fddc-vlg2f"
Jul 07 16:30:33.491556 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.491547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7679fddc-vlg2f" event={"ID":"79ea75e6-138e-475e-9306-bce919a44298","Type":"ContainerDied","Data":"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"}
Jul 07 16:30:33.491763 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.491597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7679fddc-vlg2f" event={"ID":"79ea75e6-138e-475e-9306-bce919a44298","Type":"ContainerDied","Data":"54412cb02163d9707308578c3d992fc9ac55ed2d0f5ba98b5388f6d90bd63c6e"}
Jul 07 16:30:33.491763 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.491619    2565 scope.go:117] "RemoveContainer" containerID="4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"
Jul 07 16:30:33.499887 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.499868    2565 scope.go:117] "RemoveContainer" containerID="4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"
Jul 07 16:30:33.500158 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:30:33.500138    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf\": container with ID starting with 4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf not found: ID does not exist" containerID="4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"
Jul 07 16:30:33.500219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.500171    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf"} err="failed to get container status \"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf\": rpc error: code = NotFound desc = could not find container \"4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf\": container with ID starting with 4365da40b42c101367c90877109d58583da100d84f0f2bab0e65cf14859ce9cf not found: ID does not exist"
Jul 07 16:30:33.513650 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.513620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:30:33.519147 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:33.519120    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7679fddc-vlg2f"]
Jul 07 16:30:34.093931 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:30:34.093893    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ea75e6-138e-475e-9306-bce919a44298" path="/var/lib/kubelet/pods/79ea75e6-138e-475e-9306-bce919a44298/volumes"
Jul 07 16:32:15.313323 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313239    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"]
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313581    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" containerName="console"
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313598    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" containerName="console"
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313613    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ea75e6-138e-475e-9306-bce919a44298" containerName="console"
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313621    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ea75e6-138e-475e-9306-bce919a44298" containerName="console"
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313689    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66fe313b-f2d0-4f56-a347-dfd840cd9e0a" containerName="console"
Jul 07 16:32:15.313926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.313701    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ea75e6-138e-475e-9306-bce919a44298" containerName="console"
Jul 07 16:32:15.315974 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.315953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.319152 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.319133    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:32:15.319259 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.319170    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 07 16:32:15.319436 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.319421    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rclh6\""
Jul 07 16:32:15.334529 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.334504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"]
Jul 07 16:32:15.405160 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.405134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b94bac88-5e8e-4a9a-a160-458e2558978f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.405275 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.405176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjlg\" (UniqueName: \"kubernetes.io/projected/b94bac88-5e8e-4a9a-a160-458e2558978f-kube-api-access-8tjlg\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.506157 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.506133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b94bac88-5e8e-4a9a-a160-458e2558978f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.506265 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.506175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjlg\" (UniqueName: \"kubernetes.io/projected/b94bac88-5e8e-4a9a-a160-458e2558978f-kube-api-access-8tjlg\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.506526 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.506503    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b94bac88-5e8e-4a9a-a160-458e2558978f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.514889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.514860    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjlg\" (UniqueName: \"kubernetes.io/projected/b94bac88-5e8e-4a9a-a160-458e2558978f-kube-api-access-8tjlg\") pod \"cert-manager-operator-controller-manager-55c444bf86-2h2tj\" (UID: \"b94bac88-5e8e-4a9a-a160-458e2558978f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.624522 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.624476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"
Jul 07 16:32:15.775900 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.775827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj"]
Jul 07 16:32:15.780049 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:32:15.780024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94bac88_5e8e_4a9a_a160_458e2558978f.slice/crio-5e1ee55db64ada718e57a5161e633421a65e4573b0fa1bd5e54b192c9b224e4b WatchSource:0}: Error finding container 5e1ee55db64ada718e57a5161e633421a65e4573b0fa1bd5e54b192c9b224e4b: Status 404 returned error can't find the container with id 5e1ee55db64ada718e57a5161e633421a65e4573b0fa1bd5e54b192c9b224e4b
Jul 07 16:32:15.848399 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:15.848366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj" event={"ID":"b94bac88-5e8e-4a9a-a160-458e2558978f","Type":"ContainerStarted","Data":"5e1ee55db64ada718e57a5161e633421a65e4573b0fa1bd5e54b192c9b224e4b"}
Jul 07 16:32:18.863168 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:18.863130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj" event={"ID":"b94bac88-5e8e-4a9a-a160-458e2558978f","Type":"ContainerStarted","Data":"64ec5a2d493dc27d4894b91e79998abaa8ea7f2c60911ccd03997c1525c7482c"}
Jul 07 16:32:18.889900 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:18.889857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2h2tj" podStartSLOduration=1.170296921 podStartE2EDuration="3.889843145s" totalImagesPullingTime="2.719546224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:32:18.887922462 +0000 UTC m=+537.420839120" watchObservedRunningTime="2026-07-07 16:32:18.889843145 +0000 UTC m=+537.422759789"
Jul 07 16:32:29.837550 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.837512    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4mphf"]
Jul 07 16:32:29.840010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.839989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:29.842898 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.842878    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 16:32:29.843956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.843940    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 16:32:29.844010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.843983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hbld5\""
Jul 07 16:32:29.850203 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.850180    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4mphf"]
Jul 07 16:32:29.912828 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.912804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:29.912937 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:29.912863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmpz\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-kube-api-access-tbmpz\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.013727 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.013704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmpz\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-kube-api-access-tbmpz\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.013868 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.013741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.022353 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.022325    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.022461 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.022444    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmpz\" (UniqueName: \"kubernetes.io/projected/2d12dac8-eb7f-479b-92ac-03da273d7a2b-kube-api-access-tbmpz\") pod \"cert-manager-55b4d954d5-4mphf\" (UID: \"2d12dac8-eb7f-479b-92ac-03da273d7a2b\") " pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.149187 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.149115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4mphf"
Jul 07 16:32:30.280528 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.280500    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4mphf"]
Jul 07 16:32:30.284958 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:32:30.284933    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d12dac8_eb7f_479b_92ac_03da273d7a2b.slice/crio-ad35d3ae0ec303e25bb6e728922090edbd25bda4a7b82fe10400ec144910de69 WatchSource:0}: Error finding container ad35d3ae0ec303e25bb6e728922090edbd25bda4a7b82fe10400ec144910de69: Status 404 returned error can't find the container with id ad35d3ae0ec303e25bb6e728922090edbd25bda4a7b82fe10400ec144910de69
Jul 07 16:32:30.903621 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:30.903587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4mphf" event={"ID":"2d12dac8-eb7f-479b-92ac-03da273d7a2b","Type":"ContainerStarted","Data":"ad35d3ae0ec303e25bb6e728922090edbd25bda4a7b82fe10400ec144910de69"}
Jul 07 16:32:33.915873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:33.915834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4mphf" event={"ID":"2d12dac8-eb7f-479b-92ac-03da273d7a2b","Type":"ContainerStarted","Data":"5c19c4f14a616c7aea6d95020c6dadc943519c2974a2eb09a7268bdfd20f7146"}
Jul 07 16:32:33.933668 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:33.933621    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4mphf" podStartSLOduration=2.206511442 podStartE2EDuration="4.933606164s" totalImagesPullingTime="2.727094722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:32:33.932473582 +0000 UTC m=+552.465390228" watchObservedRunningTime="2026-07-07 16:32:33.933606164 +0000 UTC m=+552.466522808"
Jul 07 16:32:46.938250 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.938218    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"]
Jul 07 16:32:46.945866 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.945840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:46.951178 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.951147    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 16:32:46.952090 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.952064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xm4t8\""
Jul 07 16:32:46.952090 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.952080    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:32:46.953517 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:46.953499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"]
Jul 07 16:32:47.044706 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.044679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/453f65f2-e43f-403f-b2ea-257c21e35e1c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.044826 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.044720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptkk\" (UniqueName: \"kubernetes.io/projected/453f65f2-e43f-403f-b2ea-257c21e35e1c-kube-api-access-kptkk\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.145902 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.145883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/453f65f2-e43f-403f-b2ea-257c21e35e1c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.145995 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.145912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptkk\" (UniqueName: \"kubernetes.io/projected/453f65f2-e43f-403f-b2ea-257c21e35e1c-kube-api-access-kptkk\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.146247 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.146230    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/453f65f2-e43f-403f-b2ea-257c21e35e1c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.155708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.155680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptkk\" (UniqueName: \"kubernetes.io/projected/453f65f2-e43f-403f-b2ea-257c21e35e1c-kube-api-access-kptkk\") pod \"openshift-lws-operator-fd8ccff4c-5nmtw\" (UID: \"453f65f2-e43f-403f-b2ea-257c21e35e1c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.260454 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.260401    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"
Jul 07 16:32:47.405596 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.405569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw"]
Jul 07 16:32:47.408622 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:32:47.408595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod453f65f2_e43f_403f_b2ea_257c21e35e1c.slice/crio-ec6bf5bb23d5fd17762cb7ad6efc1452763f7e357969dab1bb33848c65349ff7 WatchSource:0}: Error finding container ec6bf5bb23d5fd17762cb7ad6efc1452763f7e357969dab1bb33848c65349ff7: Status 404 returned error can't find the container with id ec6bf5bb23d5fd17762cb7ad6efc1452763f7e357969dab1bb33848c65349ff7
Jul 07 16:32:47.964882 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:47.964841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw" event={"ID":"453f65f2-e43f-403f-b2ea-257c21e35e1c","Type":"ContainerStarted","Data":"ec6bf5bb23d5fd17762cb7ad6efc1452763f7e357969dab1bb33848c65349ff7"}
Jul 07 16:32:50.980869 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:32:50.980832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw" event={"ID":"453f65f2-e43f-403f-b2ea-257c21e35e1c","Type":"ContainerStarted","Data":"a68e953e2aed98c7414400c8414ffb94d1bbb3047f86b3c87bb422f2c020ced8"}
Jul 07 16:33:10.408564 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.408509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5nmtw" podStartSLOduration=21.640248013 podStartE2EDuration="24.408492781s" totalImagesPullingTime="2.768244768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:32:51.001364782 +0000 UTC m=+569.534281427" watchObservedRunningTime="2026-07-07 16:33:10.408492781 +0000 UTC m=+588.941409424"
Jul 07 16:33:10.409100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.409085    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"]
Jul 07 16:33:10.411331 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.411317    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.415777 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.415750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 16:33:10.415777 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.415763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 16:33:10.419306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.419285    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9fsmk\""
Jul 07 16:33:10.419585 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.419569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 16:33:10.419662 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.419589    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 16:33:10.420875 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.420845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.420971 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.420879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n8h\" (UniqueName: \"kubernetes.io/projected/44532bf2-29ef-42b8-8a92-a5a043ba625f-kube-api-access-95n8h\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.420971 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.420945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.421089 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.421008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.421089 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.421047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.429671 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.429636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"]
Jul 07 16:33:10.521384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.521384 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.521618 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.521618 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n8h\" (UniqueName: \"kubernetes.io/projected/44532bf2-29ef-42b8-8a92-a5a043ba625f-kube-api-access-95n8h\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.521699 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.521911 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.521891    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.523708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.523687    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44532bf2-29ef-42b8-8a92-a5a043ba625f-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.523837 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.523812    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.523939 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.523924    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44532bf2-29ef-42b8-8a92-a5a043ba625f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.540827 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.540778    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n8h\" (UniqueName: \"kubernetes.io/projected/44532bf2-29ef-42b8-8a92-a5a043ba625f-kube-api-access-95n8h\") pod \"opendatahub-operator-controller-manager-7fb6d99899-6pz6b\" (UID: \"44532bf2-29ef-42b8-8a92-a5a043ba625f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.721060 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.720951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:10.857183 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:10.857159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"]
Jul 07 16:33:10.859959 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:10.859928    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44532bf2_29ef_42b8_8a92_a5a043ba625f.slice/crio-172594d1142ce44e3ac4fdd777146f096f44b27081b76f4e70bc378b0a7c4aef WatchSource:0}: Error finding container 172594d1142ce44e3ac4fdd777146f096f44b27081b76f4e70bc378b0a7c4aef: Status 404 returned error can't find the container with id 172594d1142ce44e3ac4fdd777146f096f44b27081b76f4e70bc378b0a7c4aef
Jul 07 16:33:11.056902 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:11.056816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b" event={"ID":"44532bf2-29ef-42b8-8a92-a5a043ba625f","Type":"ContainerStarted","Data":"172594d1142ce44e3ac4fdd777146f096f44b27081b76f4e70bc378b0a7c4aef"}
Jul 07 16:33:15.076083 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:15.076041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b" event={"ID":"44532bf2-29ef-42b8-8a92-a5a043ba625f","Type":"ContainerStarted","Data":"d51043c845ec0071e8eaf223d380ebef2862fa1188e47293a66bdbd7a94f4034"}
Jul 07 16:33:16.080800 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:16.080757    2565 generic.go:356] "Generic (PLEG): container finished" podID="44532bf2-29ef-42b8-8a92-a5a043ba625f" containerID="d51043c845ec0071e8eaf223d380ebef2862fa1188e47293a66bdbd7a94f4034" exitCode=0
Jul 07 16:33:16.081185 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:16.080813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b" event={"ID":"44532bf2-29ef-42b8-8a92-a5a043ba625f","Type":"ContainerDied","Data":"d51043c845ec0071e8eaf223d380ebef2862fa1188e47293a66bdbd7a94f4034"}
Jul 07 16:33:17.086634 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:17.086599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b" event={"ID":"44532bf2-29ef-42b8-8a92-a5a043ba625f","Type":"ContainerStarted","Data":"b72b50dc97aa72714ce981b56334433fe5a852df6be00a022297fede1ffae706"}
Jul 07 16:33:17.087059 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:17.086747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:17.120348 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:17.120303    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b" podStartSLOduration=2.404182095 podStartE2EDuration="7.120288103s" totalImagesPullingTime="3.688799139s" totalInitContainerRuntime="1.027306869s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:17.117839014 +0000 UTC m=+595.650755684" watchObservedRunningTime="2026-07-07 16:33:17.120288103 +0000 UTC m=+595.653204747"
Jul 07 16:33:28.094987 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:28.094960    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-6pz6b"
Jul 07 16:33:30.849295 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.849257    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"]
Jul 07 16:33:30.852278 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.852261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:30.858414 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.858398    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 16:33:30.859366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.859347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 16:33:30.859435 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.859352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 16:33:30.869867 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.869844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 16:33:30.874559 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.874532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"]
Jul 07 16:33:30.982114 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.982082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063ae709-1e25-408e-b2ac-1c21f34fefb1-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:30.982296 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.982122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ae709-1e25-408e-b2ac-1c21f34fefb1-tmp\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:30.982296 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:30.982148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btl56\" (UniqueName: \"kubernetes.io/projected/063ae709-1e25-408e-b2ac-1c21f34fefb1-kube-api-access-btl56\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.042800 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.042755    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-46hj7"]
Jul 07 16:33:31.045854 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.045829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.050119 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.050097    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 07 16:33:31.050231 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.050102    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nz8sg\""
Jul 07 16:33:31.066120 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.066093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-46hj7"]
Jul 07 16:33:31.082839 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.082807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063ae709-1e25-408e-b2ac-1c21f34fefb1-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.082973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.082855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ae709-1e25-408e-b2ac-1c21f34fefb1-tmp\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.082973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.082882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sml5\" (UniqueName: \"kubernetes.io/projected/61ccf201-2c95-45b5-a08e-7776ad2134e8-kube-api-access-9sml5\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.082973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.082910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btl56\" (UniqueName: \"kubernetes.io/projected/063ae709-1e25-408e-b2ac-1c21f34fefb1-kube-api-access-btl56\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.082973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.082953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.085316 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.085293    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063ae709-1e25-408e-b2ac-1c21f34fefb1-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.085493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.085476    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ae709-1e25-408e-b2ac-1c21f34fefb1-tmp\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.099348 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.099294    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btl56\" (UniqueName: \"kubernetes.io/projected/063ae709-1e25-408e-b2ac-1c21f34fefb1-kube-api-access-btl56\") pod \"kube-auth-proxy-6dfd697d7-5t7cd\" (UID: \"063ae709-1e25-408e-b2ac-1c21f34fefb1\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.161575 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.161547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"
Jul 07 16:33:31.184112 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.184083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sml5\" (UniqueName: \"kubernetes.io/projected/61ccf201-2c95-45b5-a08e-7776ad2134e8-kube-api-access-9sml5\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.184248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.184143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.184248 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:31.184219    2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 16:33:31.184375 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:31.184294    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert podName:61ccf201-2c95-45b5-a08e-7776ad2134e8 nodeName:}" failed. No retries permitted until 2026-07-07 16:33:31.684274588 +0000 UTC m=+610.217191217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert") pod "odh-model-controller-7998c67dc4-46hj7" (UID: "61ccf201-2c95-45b5-a08e-7776ad2134e8") : secret "odh-model-controller-webhook-cert" not found
Jul 07 16:33:31.202817 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.202774    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sml5\" (UniqueName: \"kubernetes.io/projected/61ccf201-2c95-45b5-a08e-7776ad2134e8-kube-api-access-9sml5\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.298527 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.298488    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd"]
Jul 07 16:33:31.301184 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:31.301159    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063ae709_1e25_408e_b2ac_1c21f34fefb1.slice/crio-964255cad71bc199b50c5f9ef27fef6ec16166f4638f3617cbefa151c9cfadb0 WatchSource:0}: Error finding container 964255cad71bc199b50c5f9ef27fef6ec16166f4638f3617cbefa151c9cfadb0: Status 404 returned error can't find the container with id 964255cad71bc199b50c5f9ef27fef6ec16166f4638f3617cbefa151c9cfadb0
Jul 07 16:33:31.302729 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.302717    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:33:31.688358 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.688326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.690643 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.690625    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61ccf201-2c95-45b5-a08e-7776ad2134e8-cert\") pod \"odh-model-controller-7998c67dc4-46hj7\" (UID: \"61ccf201-2c95-45b5-a08e-7776ad2134e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:31.955278 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:31.955201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:32.099994 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.099970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-46hj7"]
Jul 07 16:33:32.102893 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:32.102867    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ccf201_2c95_45b5_a08e_7776ad2134e8.slice/crio-5655fff6da01d86995ad11d984d099827d45907c6484d4a0ea1766b369fb6ac1 WatchSource:0}: Error finding container 5655fff6da01d86995ad11d984d099827d45907c6484d4a0ea1766b369fb6ac1: Status 404 returned error can't find the container with id 5655fff6da01d86995ad11d984d099827d45907c6484d4a0ea1766b369fb6ac1
Jul 07 16:33:32.146343 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.146308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7" event={"ID":"61ccf201-2c95-45b5-a08e-7776ad2134e8","Type":"ContainerStarted","Data":"5655fff6da01d86995ad11d984d099827d45907c6484d4a0ea1766b369fb6ac1"}
Jul 07 16:33:32.147491 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.147467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd" event={"ID":"063ae709-1e25-408e-b2ac-1c21f34fefb1","Type":"ContainerStarted","Data":"964255cad71bc199b50c5f9ef27fef6ec16166f4638f3617cbefa151c9cfadb0"}
Jul 07 16:33:32.312980 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.312902    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8jbcp"]
Jul 07 16:33:32.316313 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.316287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.325471 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.325446    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 07 16:33:32.328734 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.328714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8jbcp"]
Jul 07 16:33:32.393829 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.393778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv2h\" (UniqueName: \"kubernetes.io/projected/e14eda58-c8d5-47b8-9622-f57afb62be5e-kube-api-access-bvv2h\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.393979 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.393893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.495019 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.494990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv2h\" (UniqueName: \"kubernetes.io/projected/e14eda58-c8d5-47b8-9622-f57afb62be5e-kube-api-access-bvv2h\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.495161 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.495053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.495200 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:32.495164    2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 16:33:32.495245 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:32.495235    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs podName:e14eda58-c8d5-47b8-9622-f57afb62be5e nodeName:}" failed. No retries permitted until 2026-07-07 16:33:32.995220164 +0000 UTC m=+611.528136791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs") pod "model-serving-api-54d89bd5fc-8jbcp" (UID: "e14eda58-c8d5-47b8-9622-f57afb62be5e") : secret "model-serving-api-tls" not found
Jul 07 16:33:32.522665 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.522631    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv2h\" (UniqueName: \"kubernetes.io/projected/e14eda58-c8d5-47b8-9622-f57afb62be5e-kube-api-access-bvv2h\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:32.586761 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.586728    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:32.590047 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.590018    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.593145 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.593111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vmn9p\""
Jul 07 16:33:32.593278 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.593164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 16:33:32.593585 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.593564    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 16:33:32.593585 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.593579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 16:33:32.593909 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.593868    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 16:33:32.607680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.607657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:32.696807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.696753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.696807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.696809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.697007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.696890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.697007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.696932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45d4\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.697007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.696980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.697114 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.697022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.697114 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.697044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798204 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45d4\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798592 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.798592 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.798433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.799242 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.799209    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.801103 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.801071    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.801198 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.801078    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.801198 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.801159    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.801287 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.801202    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.815683 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.815640    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45d4\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.817963 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.817921    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token\") pod \"istiod-openshift-gateway-7574459656-bvkvg\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:32.902622 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:32.902575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:33.000270 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.000243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:33.002601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.002579    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14eda58-c8d5-47b8-9622-f57afb62be5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-8jbcp\" (UID: \"e14eda58-c8d5-47b8-9622-f57afb62be5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:33.032945 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.032913    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:33.036011 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:33.035980    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3f2136_28d0_487d_81e0_3d6aac7b0360.slice/crio-a81240abfa1d5c8b65c97212865b302623c12266e30800f61b51e83c176d0051 WatchSource:0}: Error finding container a81240abfa1d5c8b65c97212865b302623c12266e30800f61b51e83c176d0051: Status 404 returned error can't find the container with id a81240abfa1d5c8b65c97212865b302623c12266e30800f61b51e83c176d0051
Jul 07 16:33:33.154501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.154431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" event={"ID":"6d3f2136-28d0-487d-81e0-3d6aac7b0360","Type":"ContainerStarted","Data":"a81240abfa1d5c8b65c97212865b302623c12266e30800f61b51e83c176d0051"}
Jul 07 16:33:33.228929 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.228906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:33.392809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:33.390509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8jbcp"]
Jul 07 16:33:33.398540 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:33.398479    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14eda58_c8d5_47b8_9622_f57afb62be5e.slice/crio-4a0db73a5ee24d0159762c6f59371adf40fd1d43625f940d63635494f253b94d WatchSource:0}: Error finding container 4a0db73a5ee24d0159762c6f59371adf40fd1d43625f940d63635494f253b94d: Status 404 returned error can't find the container with id 4a0db73a5ee24d0159762c6f59371adf40fd1d43625f940d63635494f253b94d
Jul 07 16:33:34.163988 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:34.163939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp" event={"ID":"e14eda58-c8d5-47b8-9622-f57afb62be5e","Type":"ContainerStarted","Data":"4a0db73a5ee24d0159762c6f59371adf40fd1d43625f940d63635494f253b94d"}
Jul 07 16:33:37.323456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:37.323426    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:33:37.323695 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:37.323486    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:33:38.182576 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.182545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp" event={"ID":"e14eda58-c8d5-47b8-9622-f57afb62be5e","Type":"ContainerStarted","Data":"62977f51d6a5fae27cbe7af883506b20339f699a6db381131ccae30bb27999d6"}
Jul 07 16:33:38.182748 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.182607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:33:38.184156 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.184133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" event={"ID":"6d3f2136-28d0-487d-81e0-3d6aac7b0360","Type":"ContainerStarted","Data":"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"}
Jul 07 16:33:38.184281 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.184245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:38.185680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.185658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7" event={"ID":"61ccf201-2c95-45b5-a08e-7776ad2134e8","Type":"ContainerStarted","Data":"56188653ff8410b4ba0975a53ac667f9af19abe8141ddbe63b934428c03cb98e"}
Jul 07 16:33:38.185821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.185806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:38.185982 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.185962    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-bvkvg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 16:33:38.186056 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.186008    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:33:38.187163 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.187143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd" event={"ID":"063ae709-1e25-408e-b2ac-1c21f34fefb1","Type":"ContainerStarted","Data":"3b330c15f828e4b908a10807786e76c3733e9a68bc0c5455bb1d7efcb81424d1"}
Jul 07 16:33:38.205261 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.205222    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp" podStartSLOduration=2.282678499 podStartE2EDuration="6.205210054s" totalImagesPullingTime="3.922531555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:38.203167013 +0000 UTC m=+616.736083684" watchObservedRunningTime="2026-07-07 16:33:38.205210054 +0000 UTC m=+616.738126698"
Jul 07 16:33:38.222139 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.222098    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7" podStartSLOduration=2.005860665 podStartE2EDuration="7.22208479s" totalImagesPullingTime="5.216224125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:38.220989778 +0000 UTC m=+616.753906440" watchObservedRunningTime="2026-07-07 16:33:38.22208479 +0000 UTC m=+616.755001433"
Jul 07 16:33:38.243693 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.243658    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" podStartSLOduration=1.9589941149999999 podStartE2EDuration="6.243646545s" totalImagesPullingTime="4.28465243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:38.243020859 +0000 UTC m=+616.775937504" watchObservedRunningTime="2026-07-07 16:33:38.243646545 +0000 UTC m=+616.776563190"
Jul 07 16:33:38.266042 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.266005    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-5t7cd" podStartSLOduration=2.254933608 podStartE2EDuration="8.265993527s" totalImagesPullingTime="6.011059919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:38.263895596 +0000 UTC m=+616.796812265" watchObservedRunningTime="2026-07-07 16:33:38.265993527 +0000 UTC m=+616.798910170"
Jul 07 16:33:38.881083 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.881052    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"]
Jul 07 16:33:38.884970 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.884953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.890757 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.890732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-scv9z\""
Jul 07 16:33:38.912488 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.912466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"]
Jul 07 16:33:38.952376 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952484 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952484 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952595 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952595 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952595 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952595 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597zm\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-kube-api-access-597zm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952754 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:38.952754 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:38.952632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053319 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053430 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053430 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053430 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597zm\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-kube-api-access-597zm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053430 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053625 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053881 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053698    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.053948 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.053895    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.054056 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.054034    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.054508 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.054340    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.054508 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.054458    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.059815 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.056256    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.059815 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.056451    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.076566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.076539    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597zm\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-kube-api-access-597zm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.079300 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.079280    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8bf763-6f0b-42b6-9f92-6f9a4c599851-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxgxb\" (UID: \"ef8bf763-6f0b-42b6-9f92-6f9a4c599851\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.191731 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.191688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:39.194875 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.194862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:39.369572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:39.369551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"]
Jul 07 16:33:39.371111 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:39.371089    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8bf763_6f0b_42b6_9f92_6f9a4c599851.slice/crio-4838f7fdb91add078015c8148bf8b0be7d624b50c4ae9554aa0054280476cf9a WatchSource:0}: Error finding container 4838f7fdb91add078015c8148bf8b0be7d624b50c4ae9554aa0054280476cf9a: Status 404 returned error can't find the container with id 4838f7fdb91add078015c8148bf8b0be7d624b50c4ae9554aa0054280476cf9a
Jul 07 16:33:40.197658 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:40.197622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb" event={"ID":"ef8bf763-6f0b-42b6-9f92-6f9a4c599851","Type":"ContainerStarted","Data":"4838f7fdb91add078015c8148bf8b0be7d624b50c4ae9554aa0054280476cf9a"}
Jul 07 16:33:41.932807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:41.932762    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:33:41.933047 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:41.932866    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:33:41.933047 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:41.932897    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:33:42.062521 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.062488    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"]
Jul 07 16:33:42.066014 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.065993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.073261 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.073238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 07 16:33:42.077836 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.077818    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7kftk\""
Jul 07 16:33:42.134566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.134545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"]
Jul 07 16:33:42.181460 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.181407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/841928fa-7995-4140-b430-ddc279ffe2ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.181566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.181466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfhz\" (UniqueName: \"kubernetes.io/projected/841928fa-7995-4140-b430-ddc279ffe2ad-kube-api-access-5tfhz\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.203045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.203021    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"]
Jul 07 16:33:42.206452 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.206437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.209428 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.209406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb" event={"ID":"ef8bf763-6f0b-42b6-9f92-6f9a4c599851","Type":"ContainerStarted","Data":"43d89e406f5f407f5e45ea9ee8ebf71f2d33d39238e48723e2d33419fe291dbb"}
Jul 07 16:33:42.211981 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:42.211949    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-controller-manager-dockercfg-592hz\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-592hz\"" type="*v1.Secret"
Jul 07 16:33:42.212124 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:42.211996    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret"
Jul 07 16:33:42.240394 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.240370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"]
Jul 07 16:33:42.282393 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.282368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfhz\" (UniqueName: \"kubernetes.io/projected/841928fa-7995-4140-b430-ddc279ffe2ad-kube-api-access-5tfhz\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.282519 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.282403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwgn\" (UniqueName: \"kubernetes.io/projected/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-kube-api-access-qnwgn\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.282519 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.282459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/841928fa-7995-4140-b430-ddc279ffe2ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.282628 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.282511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.284821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.284797    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/841928fa-7995-4140-b430-ddc279ffe2ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.311772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.311745    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfhz\" (UniqueName: \"kubernetes.io/projected/841928fa-7995-4140-b430-ddc279ffe2ad-kube-api-access-5tfhz\") pod \"kserve-controller-manager-d6f4b65b8-cdwlb\" (UID: \"841928fa-7995-4140-b430-ddc279ffe2ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.327709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.327671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb" podStartSLOduration=1.768229976 podStartE2EDuration="4.327659574s" totalImagesPullingTime="2.559429598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:42.326711833 +0000 UTC m=+620.859628476" watchObservedRunningTime="2026-07-07 16:33:42.327659574 +0000 UTC m=+620.860576217"
Jul 07 16:33:42.376834 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.376798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:42.383910 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.383887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwgn\" (UniqueName: \"kubernetes.io/projected/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-kube-api-access-qnwgn\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.383989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.383948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.394850 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.394828    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwgn\" (UniqueName: \"kubernetes.io/projected/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-kube-api-access-qnwgn\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:42.507649 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.507546    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"]
Jul 07 16:33:42.510651 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:42.510629    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841928fa_7995_4140_b430_ddc279ffe2ad.slice/crio-f5b579388c04c7ed7f519d3361294d59c5326f323651dcc097c3c0099c26cc6e WatchSource:0}: Error finding container f5b579388c04c7ed7f519d3361294d59c5326f323651dcc097c3c0099c26cc6e: Status 404 returned error can't find the container with id f5b579388c04c7ed7f519d3361294d59c5326f323651dcc097c3c0099c26cc6e
Jul 07 16:33:42.834162 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.834129    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:42.834344 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:42.834321    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerName="discovery" containerID="cri-o://9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587" gracePeriod=30
Jul 07 16:33:43.076809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.076772    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:43.189838 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.189759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.189838 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.189816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d45d4\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190012 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.189893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190012 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.189934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190012 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.189971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190012 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.190002    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190218 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.190029    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token\") pod \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\" (UID: \"6d3f2136-28d0-487d-81e0-3d6aac7b0360\") "
Jul 07 16:33:43.190492 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.190461    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:33:43.192528 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.192490    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:33:43.192617 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.192534    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:33:43.192727 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.192622    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:33:43.192915 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.192893    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4" (OuterVolumeSpecName: "kube-api-access-d45d4") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "kube-api-access-d45d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:33:43.193041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.193019    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs" (OuterVolumeSpecName: "local-certs") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:33:43.193089 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.193058    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts" (OuterVolumeSpecName: "cacerts") pod "6d3f2136-28d0-487d-81e0-3d6aac7b0360" (UID: "6d3f2136-28d0-487d-81e0-3d6aac7b0360"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:33:43.195280 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.195260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:43.200560 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.200490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:43.215260 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.215232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb" event={"ID":"841928fa-7995-4140-b430-ddc279ffe2ad","Type":"ContainerStarted","Data":"f5b579388c04c7ed7f519d3361294d59c5326f323651dcc097c3c0099c26cc6e"}
Jul 07 16:33:43.216810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.216762    2565 generic.go:356] "Generic (PLEG): container finished" podID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerID="9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587" exitCode=0
Jul 07 16:33:43.216893 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.216818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" event={"ID":"6d3f2136-28d0-487d-81e0-3d6aac7b0360","Type":"ContainerDied","Data":"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"}
Jul 07 16:33:43.216893 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.216865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg" event={"ID":"6d3f2136-28d0-487d-81e0-3d6aac7b0360","Type":"ContainerDied","Data":"a81240abfa1d5c8b65c97212865b302623c12266e30800f61b51e83c176d0051"}
Jul 07 16:33:43.216893 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.216892    2565 scope.go:117] "RemoveContainer" containerID="9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"
Jul 07 16:33:43.217058 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.216827    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"
Jul 07 16:33:43.217286 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.217263    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:43.217995 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.217974    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxgxb"
Jul 07 16:33:43.226949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.226932    2565 scope.go:117] "RemoveContainer" containerID="9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"
Jul 07 16:33:43.227207 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:43.227183    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587\": container with ID starting with 9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587 not found: ID does not exist" containerID="9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"
Jul 07 16:33:43.227303 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.227213    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587"} err="failed to get container status \"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587\": rpc error: code = NotFound desc = could not find container \"9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587\": container with ID starting with 9ff2798f4417fe0b5c4857737726a3355c7a5adfdd199d3688e4b674092d3587 not found: ID does not exist"
Jul 07 16:33:43.246401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.246379    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:43.249935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.249915    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bvkvg"]
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290722    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-kubeconfig\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290747    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d45d4\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-kube-api-access-d45d4\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290760    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3f2136-28d0-487d-81e0-3d6aac7b0360-local-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290774    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-dns-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290804    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-csr-ca-configmap\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290819    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d3f2136-28d0-487d-81e0-3d6aac7b0360-cacerts\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.290949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.290832    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3f2136-28d0-487d-81e0-3d6aac7b0360-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:33:43.384697 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:43.384668    2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: failed to sync secret cache: timed out waiting for the condition
Jul 07 16:33:43.384846 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:33:43.384740    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert podName:0f0e39de-e6f3-4322-9a6a-ed33b56f85f7 nodeName:}" failed. No retries permitted until 2026-07-07 16:33:43.884721568 +0000 UTC m=+622.417638201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert") pod "llmisvc-controller-manager-777c6bd7d5-n8mhx" (UID: "0f0e39de-e6f3-4322-9a6a-ed33b56f85f7") : failed to sync secret cache: timed out waiting for the condition
Jul 07 16:33:43.593764 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.593736    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-592hz\""
Jul 07 16:33:43.812233 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.812210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 16:33:43.895592 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.895538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:43.897801 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:43.897762    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0e39de-e6f3-4322-9a6a-ed33b56f85f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n8mhx\" (UID: \"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:44.016717 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:44.016698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:44.100474 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:44.100434    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" path="/var/lib/kubelet/pods/6d3f2136-28d0-487d-81e0-3d6aac7b0360/volumes"
Jul 07 16:33:44.179758 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:33:44.179717    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f0e39de_e6f3_4322_9a6a_ed33b56f85f7.slice/crio-03f45b5c166c699100776c65a0b0f90c4cc29b89b1730beffbdcc953ee7dc89c WatchSource:0}: Error finding container 03f45b5c166c699100776c65a0b0f90c4cc29b89b1730beffbdcc953ee7dc89c: Status 404 returned error can't find the container with id 03f45b5c166c699100776c65a0b0f90c4cc29b89b1730beffbdcc953ee7dc89c
Jul 07 16:33:44.187224 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:44.187200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"]
Jul 07 16:33:44.223324 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:44.223300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx" event={"ID":"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7","Type":"ContainerStarted","Data":"03f45b5c166c699100776c65a0b0f90c4cc29b89b1730beffbdcc953ee7dc89c"}
Jul 07 16:33:46.233935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:46.233894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb" event={"ID":"841928fa-7995-4140-b430-ddc279ffe2ad","Type":"ContainerStarted","Data":"2bb0c63a7f246c5a9dd49d8845dcf2a760809f8bea3a38e717a1629185564e8d"}
Jul 07 16:33:46.234330 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:46.234017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:33:46.258843 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:46.258797    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb" podStartSLOduration=1.502102405 podStartE2EDuration="4.258769323s" totalImagesPullingTime="2.756666918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:46.257430285 +0000 UTC m=+624.790346952" watchObservedRunningTime="2026-07-07 16:33:46.258769323 +0000 UTC m=+624.791685967"
Jul 07 16:33:48.243714 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:48.243659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx" event={"ID":"0f0e39de-e6f3-4322-9a6a-ed33b56f85f7","Type":"ContainerStarted","Data":"ea2a544c72c4a395f16a70475b55623eaaa2c719f354fc6289b3caa48bd95959"}
Jul 07 16:33:48.244051 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:48.243778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:33:48.271122 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:48.271076    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx" podStartSLOduration=2.470693759 podStartE2EDuration="6.271058037s" totalImagesPullingTime="3.800364278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:48.269655002 +0000 UTC m=+626.802571664" watchObservedRunningTime="2026-07-07 16:33:48.271058037 +0000 UTC m=+626.803974684"
Jul 07 16:33:49.192894 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:49.192862    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-46hj7"
Jul 07 16:33:49.194903 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:33:49.194881    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8jbcp"
Jul 07 16:34:17.243962 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:17.243928    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cdwlb"
Jul 07 16:34:19.250339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:19.250311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n8mhx"
Jul 07 16:34:32.289276 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.289235    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:32.289706 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.289532    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerName="discovery"
Jul 07 16:34:32.289706 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.289543    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerName="discovery"
Jul 07 16:34:32.289706 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.289607    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3f2136-28d0-487d-81e0-3d6aac7b0360" containerName="discovery"
Jul 07 16:34:32.295059 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.295040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:32.299708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.299676    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 16:34:32.299942 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.299915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9gr7c\""
Jul 07 16:34:32.302725 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.302702    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 16:34:32.339868 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.339846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:32.359653 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.359626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnc79\" (UniqueName: \"kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79\") pod \"kuadrant-operator-catalog-qv29x\" (UID: \"3d036f67-24bd-495c-851a-5f520506e4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:32.460206 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.460171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnc79\" (UniqueName: \"kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79\") pod \"kuadrant-operator-catalog-qv29x\" (UID: \"3d036f67-24bd-495c-851a-5f520506e4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:32.471030 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.471012    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnc79\" (UniqueName: \"kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79\") pod \"kuadrant-operator-catalog-qv29x\" (UID: \"3d036f67-24bd-495c-851a-5f520506e4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:32.597116 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.597096    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:32.597322 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.597293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:32.720544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.720517    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:32.723243 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:34:32.723218    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d036f67_24bd_495c_851a_5f520506e4ab.slice/crio-37bb04e766c46fe5769c125dd93a58097b00f42b33e6f3a5d3a4b0b2b3b50245 WatchSource:0}: Error finding container 37bb04e766c46fe5769c125dd93a58097b00f42b33e6f3a5d3a4b0b2b3b50245: Status 404 returned error can't find the container with id 37bb04e766c46fe5769c125dd93a58097b00f42b33e6f3a5d3a4b0b2b3b50245
Jul 07 16:34:32.809547 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.809523    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pxfjk"]
Jul 07 16:34:32.817797 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.817754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:32.822933 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.822913    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pxfjk"]
Jul 07 16:34:32.864055 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.864007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxccl\" (UniqueName: \"kubernetes.io/projected/bb5426e9-c3ed-4431-b338-9ccfcb77a04d-kube-api-access-nxccl\") pod \"kuadrant-operator-catalog-pxfjk\" (UID: \"bb5426e9-c3ed-4431-b338-9ccfcb77a04d\") " pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:32.964746 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.964724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxccl\" (UniqueName: \"kubernetes.io/projected/bb5426e9-c3ed-4431-b338-9ccfcb77a04d-kube-api-access-nxccl\") pod \"kuadrant-operator-catalog-pxfjk\" (UID: \"bb5426e9-c3ed-4431-b338-9ccfcb77a04d\") " pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:32.973769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:32.973743    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxccl\" (UniqueName: \"kubernetes.io/projected/bb5426e9-c3ed-4431-b338-9ccfcb77a04d-kube-api-access-nxccl\") pod \"kuadrant-operator-catalog-pxfjk\" (UID: \"bb5426e9-c3ed-4431-b338-9ccfcb77a04d\") " pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:33.127692 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:33.127637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:33.256600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:33.256577    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pxfjk"]
Jul 07 16:34:33.258405 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:34:33.258377    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5426e9_c3ed_4431_b338_9ccfcb77a04d.slice/crio-6704601e30af4d37cdecb5c3c3eb0690bac5590f400c0921bd8a1dacdc3c518f WatchSource:0}: Error finding container 6704601e30af4d37cdecb5c3c3eb0690bac5590f400c0921bd8a1dacdc3c518f: Status 404 returned error can't find the container with id 6704601e30af4d37cdecb5c3c3eb0690bac5590f400c0921bd8a1dacdc3c518f
Jul 07 16:34:33.429404 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:33.429318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk" event={"ID":"bb5426e9-c3ed-4431-b338-9ccfcb77a04d","Type":"ContainerStarted","Data":"6704601e30af4d37cdecb5c3c3eb0690bac5590f400c0921bd8a1dacdc3c518f"}
Jul 07 16:34:33.430607 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:33.430574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" event={"ID":"3d036f67-24bd-495c-851a-5f520506e4ab","Type":"ContainerStarted","Data":"37bb04e766c46fe5769c125dd93a58097b00f42b33e6f3a5d3a4b0b2b3b50245"}
Jul 07 16:34:35.441387 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.441353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk" event={"ID":"bb5426e9-c3ed-4431-b338-9ccfcb77a04d","Type":"ContainerStarted","Data":"c9fcb4fe5ff628957ddb18b4b2783eb217051f7925cc05fb840ef7505ca82fda"}
Jul 07 16:34:35.442807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.442771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" event={"ID":"3d036f67-24bd-495c-851a-5f520506e4ab","Type":"ContainerStarted","Data":"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"}
Jul 07 16:34:35.442932 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.442860    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" podUID="3d036f67-24bd-495c-851a-5f520506e4ab" containerName="registry-server" containerID="cri-o://419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9" gracePeriod=2
Jul 07 16:34:35.460741 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.460703    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk" podStartSLOduration=1.8699421630000002 podStartE2EDuration="3.460688978s" totalImagesPullingTime="1.590746815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:34:35.459978159 +0000 UTC m=+673.992894823" watchObservedRunningTime="2026-07-07 16:34:35.460688978 +0000 UTC m=+673.993605685"
Jul 07 16:34:35.479462 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.479418    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" podStartSLOduration=1.357012292 podStartE2EDuration="3.479407048s" totalImagesPullingTime="2.122394756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:34:35.477879306 +0000 UTC m=+674.010795949" watchObservedRunningTime="2026-07-07 16:34:35.479407048 +0000 UTC m=+674.012323692"
Jul 07 16:34:35.682048 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.682021    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:35.789556 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.789475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnc79\" (UniqueName: \"kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79\") pod \"3d036f67-24bd-495c-851a-5f520506e4ab\" (UID: \"3d036f67-24bd-495c-851a-5f520506e4ab\") "
Jul 07 16:34:35.791537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.791511    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79" (OuterVolumeSpecName: "kube-api-access-bnc79") pod "3d036f67-24bd-495c-851a-5f520506e4ab" (UID: "3d036f67-24bd-495c-851a-5f520506e4ab"). InnerVolumeSpecName "kube-api-access-bnc79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:34:35.890978 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:35.890938    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnc79\" (UniqueName: \"kubernetes.io/projected/3d036f67-24bd-495c-851a-5f520506e4ab-kube-api-access-bnc79\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:34:36.448653 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.448566    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d036f67-24bd-495c-851a-5f520506e4ab" containerID="419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9" exitCode=0
Jul 07 16:34:36.448653 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.448610    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qv29x"
Jul 07 16:34:36.449183 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.448652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" event={"ID":"3d036f67-24bd-495c-851a-5f520506e4ab","Type":"ContainerDied","Data":"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"}
Jul 07 16:34:36.449183 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.448696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qv29x" event={"ID":"3d036f67-24bd-495c-851a-5f520506e4ab","Type":"ContainerDied","Data":"37bb04e766c46fe5769c125dd93a58097b00f42b33e6f3a5d3a4b0b2b3b50245"}
Jul 07 16:34:36.449183 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.448720    2565 scope.go:117] "RemoveContainer" containerID="419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"
Jul 07 16:34:36.457248 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.457232    2565 scope.go:117] "RemoveContainer" containerID="419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"
Jul 07 16:34:36.457503 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:34:36.457487    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9\": container with ID starting with 419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9 not found: ID does not exist" containerID="419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"
Jul 07 16:34:36.457543 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.457511    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9"} err="failed to get container status \"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9\": rpc error: code = NotFound desc = could not find container \"419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9\": container with ID starting with 419fbb7bcfcbf3cc4711d7ff18c74df32b40e33e2b08a3123fd81be94bfa1fc9 not found: ID does not exist"
Jul 07 16:34:36.465281 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.465259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:36.469567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:36.469549    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qv29x"]
Jul 07 16:34:38.095090 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:38.095053    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d036f67-24bd-495c-851a-5f520506e4ab" path="/var/lib/kubelet/pods/3d036f67-24bd-495c-851a-5f520506e4ab/volumes"
Jul 07 16:34:43.128176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:43.128138    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:43.128608 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:43.128204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:43.152019 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:43.151987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:34:43.509078 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:34:43.509011    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-pxfjk"
Jul 07 16:35:03.823373 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.823338    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"]
Jul 07 16:35:03.823852 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.823664    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d036f67-24bd-495c-851a-5f520506e4ab" containerName="registry-server"
Jul 07 16:35:03.823852 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.823675    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d036f67-24bd-495c-851a-5f520506e4ab" containerName="registry-server"
Jul 07 16:35:03.823852 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.823733    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d036f67-24bd-495c-851a-5f520506e4ab" containerName="registry-server"
Jul 07 16:35:03.827562 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.827541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:03.830467 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.830439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-88nfz\""
Jul 07 16:35:03.842919 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:03.842895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"]
Jul 07 16:35:04.005409 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.005382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdgl\" (UniqueName: \"kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl\") pod \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" (UID: \"6efeca3b-e653-479f-b519-9acb45cb066e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:04.106734 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.106674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdgl\" (UniqueName: \"kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl\") pod \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" (UID: \"6efeca3b-e653-479f-b519-9acb45cb066e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:04.116299 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.116274    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdgl\" (UniqueName: \"kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl\") pod \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" (UID: \"6efeca3b-e653-479f-b519-9acb45cb066e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:04.137191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.137165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:04.262464 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.262434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"]
Jul 07 16:35:04.265734 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:04.265710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efeca3b_e653_479f_b519_9acb45cb066e.slice/crio-1bf5bca46be3ced6fccd0ef134cc6b138bdd6ca7f7f4f076132731c69a93ce93 WatchSource:0}: Error finding container 1bf5bca46be3ced6fccd0ef134cc6b138bdd6ca7f7f4f076132731c69a93ce93: Status 404 returned error can't find the container with id 1bf5bca46be3ced6fccd0ef134cc6b138bdd6ca7f7f4f076132731c69a93ce93
Jul 07 16:35:04.576528 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:04.576500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn" event={"ID":"6efeca3b-e653-479f-b519-9acb45cb066e","Type":"ContainerStarted","Data":"1bf5bca46be3ced6fccd0ef134cc6b138bdd6ca7f7f4f076132731c69a93ce93"}
Jul 07 16:35:06.586184 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:06.586148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn" event={"ID":"6efeca3b-e653-479f-b519-9acb45cb066e","Type":"ContainerStarted","Data":"86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0"}
Jul 07 16:35:06.586525 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:06.586216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:06.605364 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:06.605315    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn" podStartSLOduration=1.7051044800000001 podStartE2EDuration="3.605300504s" totalImagesPullingTime="1.900196024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:06.604156553 +0000 UTC m=+705.137073222" watchObservedRunningTime="2026-07-07 16:35:06.605300504 +0000 UTC m=+705.138217147"
Jul 07 16:35:07.309702 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.309672    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"]
Jul 07 16:35:07.312475 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.312458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:07.315620 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.315601    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-msjxr\""
Jul 07 16:35:07.318401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.318383    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 16:35:07.337749 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.337729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"]
Jul 07 16:35:07.433037 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.433011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psnk8\" (UniqueName: \"kubernetes.io/projected/dee4984f-5355-4343-a1e3-8c670bc29588-kube-api-access-psnk8\") pod \"dns-operator-controller-manager-864dd78c4c-tp25f\" (UID: \"dee4984f-5355-4343-a1e3-8c670bc29588\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:07.533816 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.533773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psnk8\" (UniqueName: \"kubernetes.io/projected/dee4984f-5355-4343-a1e3-8c670bc29588-kube-api-access-psnk8\") pod \"dns-operator-controller-manager-864dd78c4c-tp25f\" (UID: \"dee4984f-5355-4343-a1e3-8c670bc29588\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:07.551804 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.551759    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psnk8\" (UniqueName: \"kubernetes.io/projected/dee4984f-5355-4343-a1e3-8c670bc29588-kube-api-access-psnk8\") pod \"dns-operator-controller-manager-864dd78c4c-tp25f\" (UID: \"dee4984f-5355-4343-a1e3-8c670bc29588\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:07.622670 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.622639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:07.769743 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:07.769721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"]
Jul 07 16:35:07.772250 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:07.772219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee4984f_5355_4343_a1e3_8c670bc29588.slice/crio-dc71759585c171e6b3e3444df1683ffc2ff578da51e91baf53fc93b6baf4d7e2 WatchSource:0}: Error finding container dc71759585c171e6b3e3444df1683ffc2ff578da51e91baf53fc93b6baf4d7e2: Status 404 returned error can't find the container with id dc71759585c171e6b3e3444df1683ffc2ff578da51e91baf53fc93b6baf4d7e2
Jul 07 16:35:08.595709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:08.595674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f" event={"ID":"dee4984f-5355-4343-a1e3-8c670bc29588","Type":"ContainerStarted","Data":"dc71759585c171e6b3e3444df1683ffc2ff578da51e91baf53fc93b6baf4d7e2"}
Jul 07 16:35:10.607122 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:10.607077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f" event={"ID":"dee4984f-5355-4343-a1e3-8c670bc29588","Type":"ContainerStarted","Data":"3709a6708665604bf143d52723d567af8c3fe3c222bc2f838ed99900398dd00f"}
Jul 07 16:35:10.607558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:10.607235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:10.635562 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:10.635512    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f" podStartSLOduration=1.516478539 podStartE2EDuration="3.635495445s" totalImagesPullingTime="2.119016906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:10.634426874 +0000 UTC m=+709.167343518" watchObservedRunningTime="2026-07-07 16:35:10.635495445 +0000 UTC m=+709.168412169"
Jul 07 16:35:12.272704 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.272666    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"]
Jul 07 16:35:12.275228 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.275194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.277433 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.277413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9jd\" (UniqueName: \"kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.277527 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.277451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.279187 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.279168    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gpg2n\""
Jul 07 16:35:12.295290 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.295262    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"]
Jul 07 16:35:12.377903 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.377879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.378006 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.377963    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9jd\" (UniqueName: \"kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.378300 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.378278    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.398574 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.398555    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9jd\" (UniqueName: \"kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.585082 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.585061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:12.715923 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:12.715893    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"]
Jul 07 16:35:12.719019 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:12.718988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc159a3d4_f95d_411c_ac33_573e5ac3a9ad.slice/crio-6aaaabbe935ab7a52064522a92314899ef59c4d312f7f46eb569d2fd831d509a WatchSource:0}: Error finding container 6aaaabbe935ab7a52064522a92314899ef59c4d312f7f46eb569d2fd831d509a: Status 404 returned error can't find the container with id 6aaaabbe935ab7a52064522a92314899ef59c4d312f7f46eb569d2fd831d509a
Jul 07 16:35:13.622961 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:13.622883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx" event={"ID":"c159a3d4-f95d-411c-ac33-573e5ac3a9ad","Type":"ContainerStarted","Data":"6aaaabbe935ab7a52064522a92314899ef59c4d312f7f46eb569d2fd831d509a"}
Jul 07 16:35:16.011097 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.011063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"]
Jul 07 16:35:16.013532 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.013515    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:16.016006 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.015988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f68k6\""
Jul 07 16:35:16.025360 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.025338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"]
Jul 07 16:35:16.103616 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.103586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjb47\" (UniqueName: \"kubernetes.io/projected/2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17-kube-api-access-fjb47\") pod \"authorino-operator-75cc8d5cb7-bs26p\" (UID: \"2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:16.204591 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.204565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjb47\" (UniqueName: \"kubernetes.io/projected/2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17-kube-api-access-fjb47\") pod \"authorino-operator-75cc8d5cb7-bs26p\" (UID: \"2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:16.213521 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.213496    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjb47\" (UniqueName: \"kubernetes.io/projected/2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17-kube-api-access-fjb47\") pod \"authorino-operator-75cc8d5cb7-bs26p\" (UID: \"2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:16.323583 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.323561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:16.452466 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.452444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"]
Jul 07 16:35:16.454748 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:16.454719    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f31b8a0_1741_4fa6_b7c1_7a80d63b5a17.slice/crio-b391bbb5a79de1d9c0f4b5331ed24aa6aff5cbfd06d061b03c4687378ef65fd7 WatchSource:0}: Error finding container b391bbb5a79de1d9c0f4b5331ed24aa6aff5cbfd06d061b03c4687378ef65fd7: Status 404 returned error can't find the container with id b391bbb5a79de1d9c0f4b5331ed24aa6aff5cbfd06d061b03c4687378ef65fd7
Jul 07 16:35:16.638221 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:16.638165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p" event={"ID":"2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17","Type":"ContainerStarted","Data":"b391bbb5a79de1d9c0f4b5331ed24aa6aff5cbfd06d061b03c4687378ef65fd7"}
Jul 07 16:35:17.592708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:17.592677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:20.659632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.659594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx" event={"ID":"c159a3d4-f95d-411c-ac33-573e5ac3a9ad","Type":"ContainerStarted","Data":"7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b"}
Jul 07 16:35:20.660101 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.659768    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:20.661390 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.661366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p" event={"ID":"2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17","Type":"ContainerStarted","Data":"7b30a0efaaad88c81dd4757b0803f55402034852c87b8d6b5a6a7b25aa0881f7"}
Jul 07 16:35:20.661513 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.661450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:20.696709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.696669    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx" podStartSLOduration=1.6062314880000002 podStartE2EDuration="8.69665725s" totalImagesPullingTime="7.090425762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:20.69396713 +0000 UTC m=+719.226883774" watchObservedRunningTime="2026-07-07 16:35:20.69665725 +0000 UTC m=+719.229573893"
Jul 07 16:35:20.712646 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:20.712609    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p" podStartSLOduration=2.367132291 podStartE2EDuration="5.712596409s" totalImagesPullingTime="3.345464118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:20.710766489 +0000 UTC m=+719.243683145" watchObservedRunningTime="2026-07-07 16:35:20.712596409 +0000 UTC m=+719.245513115"
Jul 07 16:35:21.615553 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:21.615520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tp25f"
Jul 07 16:35:31.667866 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:31.667829    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:31.668237 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:31.668065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bs26p"
Jul 07 16:35:33.545858 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.545829    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"]
Jul 07 16:35:33.549271 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.549254    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.596301 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.596274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"]
Jul 07 16:35:33.639459 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.639431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.639567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.639473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cst\" (UniqueName: \"kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.740382 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.740355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.740491 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.740388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cst\" (UniqueName: \"kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.740699 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.740683    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.756106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.756078    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cst\" (UniqueName: \"kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst\") pod \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.800864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.800814    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"]
Jul 07 16:35:33.801007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.800997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:33.818214 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.818195    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"]
Jul 07 16:35:33.853050 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.853019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"]
Jul 07 16:35:33.853219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.853183    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" containerName="manager" containerID="cri-o://7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b" gracePeriod=2
Jul 07 16:35:33.874144 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.874119    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"]
Jul 07 16:35:33.887070 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.887048    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"]
Jul 07 16:35:33.887348 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.887241    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" containerName="manager" containerID="cri-o://86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0" gracePeriod=2
Jul 07 16:35:33.897146 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.897126    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"]
Jul 07 16:35:33.897566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.897534    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" containerName="manager"
Jul 07 16:35:33.897566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.897554    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" containerName="manager"
Jul 07 16:35:33.897725 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.897657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" containerName="manager"
Jul 07 16:35:33.899439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.899423    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"]
Jul 07 16:35:33.899539 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.899508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:33.905613 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:33.905590    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:33.909226 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:33.909204    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:33.912011 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:33.911980    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:33.941950 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.941926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:33.942034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.941990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdz2s\" (UniqueName: \"kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:33.988883 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.988854    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"]
Jul 07 16:35:33.989293 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.989274    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" containerName="manager"
Jul 07 16:35:33.989398 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.989296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" containerName="manager"
Jul 07 16:35:33.989398 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.989391    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" containerName="manager"
Jul 07 16:35:33.992160 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.992145    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:33.992285 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.992267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:33.994477 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:33.994449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:33.995019 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:33.994984    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:33.997328 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:33.997304    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:34.013120 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.013094    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"]
Jul 07 16:35:34.013356 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:34.013332    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-vdz2s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw" podUID="e8cbce17-8d6b-4456-8982-81a21e9bdd9f"
Jul 07 16:35:34.034939 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.034236    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:35:34.037660 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.037632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:34.037810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.037745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.037879 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.037812    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"]
Jul 07 16:35:34.042823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.042507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.042823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.042572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.042823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.042603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdz2s\" (UniqueName: \"kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.042823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.042653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb6g\" (UniqueName: \"kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.042823 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.042681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz7j\" (UniqueName: \"kubernetes.io/projected/241ee8c1-0e6e-4cc3-aeca-634a1ebacab6-kube-api-access-wsz7j\") pod \"limitador-operator-controller-manager-859d7b78bd-phvsn\" (UID: \"241ee8c1-0e6e-4cc3-aeca-634a1ebacab6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:34.043439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.043392    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.059096 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:34.059033    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fbdbn\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:34.060579 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.060560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:35:34.062233 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:34.062196    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2mxvx\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:34.097005 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.096974    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"]
Jul 07 16:35:34.105104 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.105080    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdz2s\" (UniqueName: \"kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s\") pod \"kuadrant-operator-controller-manager-55467d9f7-8vcdw\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.132135 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.132034    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:34.137608 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.137590    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:34.143889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.143872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.143956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.143913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb6g\" (UniqueName: \"kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.143956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.143932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz7j\" (UniqueName: \"kubernetes.io/projected/241ee8c1-0e6e-4cc3-aeca-634a1ebacab6-kube-api-access-wsz7j\") pod \"limitador-operator-controller-manager-859d7b78bd-phvsn\" (UID: \"241ee8c1-0e6e-4cc3-aeca-634a1ebacab6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:34.144095 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.144071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.144157 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.144124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvjr\" (UniqueName: \"kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.144244 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.144230    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.166735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.166704    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz7j\" (UniqueName: \"kubernetes.io/projected/241ee8c1-0e6e-4cc3-aeca-634a1ebacab6-kube-api-access-wsz7j\") pod \"limitador-operator-controller-manager-859d7b78bd-phvsn\" (UID: \"241ee8c1-0e6e-4cc3-aeca-634a1ebacab6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:34.166735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.166719    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb6g\" (UniqueName: \"kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jn7g7\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.244622 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.244603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume\") pod \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") "
Jul 07 16:35:34.244745 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.244727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v9jd\" (UniqueName: \"kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd\") pod \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\" (UID: \"c159a3d4-f95d-411c-ac33-573e5ac3a9ad\") "
Jul 07 16:35:34.244805 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.244763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvdgl\" (UniqueName: \"kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl\") pod \"6efeca3b-e653-479f-b519-9acb45cb066e\" (UID: \"6efeca3b-e653-479f-b519-9acb45cb066e\") "
Jul 07 16:35:34.244889 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.244870    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c159a3d4-f95d-411c-ac33-573e5ac3a9ad" (UID: "c159a3d4-f95d-411c-ac33-573e5ac3a9ad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:35:34.244957 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.244940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvjr\" (UniqueName: \"kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.245097 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.245083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.245154 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.245119    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-extensions-socket-volume\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:34.245364 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.245349    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.246424 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.246404    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd" (OuterVolumeSpecName: "kube-api-access-5v9jd") pod "c159a3d4-f95d-411c-ac33-573e5ac3a9ad" (UID: "c159a3d4-f95d-411c-ac33-573e5ac3a9ad"). InnerVolumeSpecName "kube-api-access-5v9jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:35:34.246528 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.246507    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl" (OuterVolumeSpecName: "kube-api-access-gvdgl") pod "6efeca3b-e653-479f-b519-9acb45cb066e" (UID: "6efeca3b-e653-479f-b519-9acb45cb066e"). InnerVolumeSpecName "kube-api-access-gvdgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:35:34.255755 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.255737    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvjr\" (UniqueName: \"kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr\") pod \"kuadrant-operator-controller-manager-86878c769f-p9pnt\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.308306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.308282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:34.316938 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.316919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.346435 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.346404    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9jd\" (UniqueName: \"kubernetes.io/projected/c159a3d4-f95d-411c-ac33-573e5ac3a9ad-kube-api-access-5v9jd\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:34.346435 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.346432    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdgl\" (UniqueName: \"kubernetes.io/projected/6efeca3b-e653-479f-b519-9acb45cb066e-kube-api-access-gvdgl\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:34.351353 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.351325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.463208 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.463179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"]
Jul 07 16:35:34.465313 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:34.465287    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241ee8c1_0e6e_4cc3_aeca_634a1ebacab6.slice/crio-e35ce4a825d02d61f18eede2144b4f5653ffc454ede7e8d9db1482ba00c2438b WatchSource:0}: Error finding container e35ce4a825d02d61f18eede2144b4f5653ffc454ede7e8d9db1482ba00c2438b: Status 404 returned error can't find the container with id e35ce4a825d02d61f18eede2144b4f5653ffc454ede7e8d9db1482ba00c2438b
Jul 07 16:35:34.494037 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.494015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:34.495834 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:34.495807    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9250921e_441b_458e_8289_315a544d54b6.slice/crio-22f1eeac5ce208964f1bac29053fe2de615a405f0c4bd507356c62ab6e585875 WatchSource:0}: Error finding container 22f1eeac5ce208964f1bac29053fe2de615a405f0c4bd507356c62ab6e585875: Status 404 returned error can't find the container with id 22f1eeac5ce208964f1bac29053fe2de615a405f0c4bd507356c62ab6e585875
Jul 07 16:35:34.513148 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.513120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:35:34.513687 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:35:34.513661    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4a8b98_ff21_4ab4_a723_3f77d8750b7a.slice/crio-2d54a44fc9eb504fd6cdc571f7455126ab45bd25f28e8afff84a16fb69e8c174 WatchSource:0}: Error finding container 2d54a44fc9eb504fd6cdc571f7455126ab45bd25f28e8afff84a16fb69e8c174: Status 404 returned error can't find the container with id 2d54a44fc9eb504fd6cdc571f7455126ab45bd25f28e8afff84a16fb69e8c174
Jul 07 16:35:34.728165 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.728124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" event={"ID":"9250921e-441b-458e-8289-315a544d54b6","Type":"ContainerStarted","Data":"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"}
Jul 07 16:35:34.728807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.728189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" event={"ID":"9250921e-441b-458e-8289-315a544d54b6","Type":"ContainerStarted","Data":"22f1eeac5ce208964f1bac29053fe2de615a405f0c4bd507356c62ab6e585875"}
Jul 07 16:35:34.728807 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.728326    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:34.729633 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.729606    2565 generic.go:356] "Generic (PLEG): container finished" podID="6efeca3b-e653-479f-b519-9acb45cb066e" containerID="86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0" exitCode=0
Jul 07 16:35:34.729772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.729665    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbdbn"
Jul 07 16:35:34.729772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.729709    2565 scope.go:117] "RemoveContainer" containerID="86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0"
Jul 07 16:35:34.731394 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.731370    2565 generic.go:356] "Generic (PLEG): container finished" podID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" containerID="7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b" exitCode=0
Jul 07 16:35:34.731501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.731411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2mxvx"
Jul 07 16:35:34.733645 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.733487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn" event={"ID":"241ee8c1-0e6e-4cc3-aeca-634a1ebacab6","Type":"ContainerStarted","Data":"2dbd244ccf07e13040ff3d213897949903d902c9e56ba3fff388f2f60a103f3a"}
Jul 07 16:35:34.733645 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.733536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn" event={"ID":"241ee8c1-0e6e-4cc3-aeca-634a1ebacab6","Type":"ContainerStarted","Data":"e35ce4a825d02d61f18eede2144b4f5653ffc454ede7e8d9db1482ba00c2438b"}
Jul 07 16:35:34.733645 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.733590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:34.735507 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.735443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.735622 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.735470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" event={"ID":"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a","Type":"ContainerStarted","Data":"f694ab38a1a655b171385b8e718653645a12b315c92784c0bd826da2fec504cd"}
Jul 07 16:35:34.735697 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.735636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" event={"ID":"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a","Type":"ContainerStarted","Data":"2d54a44fc9eb504fd6cdc571f7455126ab45bd25f28e8afff84a16fb69e8c174"}
Jul 07 16:35:34.735753 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.735700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:34.739365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.739343    2565 scope.go:117] "RemoveContainer" containerID="86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0"
Jul 07 16:35:34.739617 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:34.739598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0\": container with ID starting with 86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0 not found: ID does not exist" containerID="86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0"
Jul 07 16:35:34.739708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.739624    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0"} err="failed to get container status \"86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0\": rpc error: code = NotFound desc = could not find container \"86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0\": container with ID starting with 86f7c073a68767c89c17164a9f8e75c84fcf43799785fda9abd62e968846b5d0 not found: ID does not exist"
Jul 07 16:35:34.739708 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.739641    2565 scope.go:117] "RemoveContainer" containerID="7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b"
Jul 07 16:35:34.744341 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.744324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:34.747856 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.747841    2565 scope.go:117] "RemoveContainer" containerID="7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b"
Jul 07 16:35:34.748099 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:34.748082    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b\": container with ID starting with 7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b not found: ID does not exist" containerID="7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b"
Jul 07 16:35:34.748148 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.748105    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b"} err="failed to get container status \"7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b\": rpc error: code = NotFound desc = could not find container \"7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b\": container with ID starting with 7a2cb87aade4b750f72af5db4dff82d194ac315adf45caab2a6efbb8b78e039b not found: ID does not exist"
Jul 07 16:35:34.761085 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.761040    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" podStartSLOduration=1.761024735 podStartE2EDuration="1.761024735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:34.757462002 +0000 UTC m=+733.290378645" watchObservedRunningTime="2026-07-07 16:35:34.761024735 +0000 UTC m=+733.293941380"
Jul 07 16:35:34.790905 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.790873    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn" podStartSLOduration=1.790863284 podStartE2EDuration="1.790863284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:34.788712723 +0000 UTC m=+733.321629379" watchObservedRunningTime="2026-07-07 16:35:34.790863284 +0000 UTC m=+733.323779928"
Jul 07 16:35:34.813651 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.813609    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" podStartSLOduration=1.813593078 podStartE2EDuration="1.813593078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:34.812729923 +0000 UTC m=+733.345646587" watchObservedRunningTime="2026-07-07 16:35:34.813593078 +0000 UTC m=+733.346509725"
Jul 07 16:35:34.851054 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.851026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume\") pod \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") "
Jul 07 16:35:34.851176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.851136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdz2s\" (UniqueName: \"kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s\") pod \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\" (UID: \"e8cbce17-8d6b-4456-8982-81a21e9bdd9f\") "
Jul 07 16:35:34.851289 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.851256    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8cbce17-8d6b-4456-8982-81a21e9bdd9f" (UID: "e8cbce17-8d6b-4456-8982-81a21e9bdd9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:35:34.851416 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.851386    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-extensions-socket-volume\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:34.853167 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.853148    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s" (OuterVolumeSpecName: "kube-api-access-vdz2s") pod "e8cbce17-8d6b-4456-8982-81a21e9bdd9f" (UID: "e8cbce17-8d6b-4456-8982-81a21e9bdd9f"). InnerVolumeSpecName "kube-api-access-vdz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:35:34.952233 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:34.952206    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdz2s\" (UniqueName: \"kubernetes.io/projected/e8cbce17-8d6b-4456-8982-81a21e9bdd9f-kube-api-access-vdz2s\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:35.741310 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:35.741283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"
Jul 07 16:35:35.792760 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:35.792731    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"]
Jul 07 16:35:35.809108 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:35.809085    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8vcdw"]
Jul 07 16:35:36.096165 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.096136    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6efeca3b-e653-479f-b519-9acb45cb066e" path="/var/lib/kubelet/pods/6efeca3b-e653-479f-b519-9acb45cb066e/volumes"
Jul 07 16:35:36.096507 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.096493    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c159a3d4-f95d-411c-ac33-573e5ac3a9ad" path="/var/lib/kubelet/pods/c159a3d4-f95d-411c-ac33-573e5ac3a9ad/volumes"
Jul 07 16:35:36.096810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.096780    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cbce17-8d6b-4456-8982-81a21e9bdd9f" path="/var/lib/kubelet/pods/e8cbce17-8d6b-4456-8982-81a21e9bdd9f/volumes"
Jul 07 16:35:36.342342 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:36.342308    2565 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-cgt6c_kuadrant-system_a2f813d1-e334-46f2-adc5-d8d7d275d880_0(85313198fe64719642e4ba6a23927f0bed7c2eed1e5c8bda1c557d42075b5c8a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-cgt6c to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 07 16:35:36.342431 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:36.342368    2565 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-cgt6c_kuadrant-system_a2f813d1-e334-46f2-adc5-d8d7d275d880_0(85313198fe64719642e4ba6a23927f0bed7c2eed1e5c8bda1c557d42075b5c8a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-cgt6c to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:36.745813 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.745768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:36.754486 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.754461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:36.756965 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:36.756937    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="a2f813d1-e334-46f2-adc5-d8d7d275d880" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:36.868886 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.868860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume\") pod \"a2f813d1-e334-46f2-adc5-d8d7d275d880\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") "
Jul 07 16:35:36.868978 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.868897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6cst\" (UniqueName: \"kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst\") pod \"a2f813d1-e334-46f2-adc5-d8d7d275d880\" (UID: \"a2f813d1-e334-46f2-adc5-d8d7d275d880\") "
Jul 07 16:35:36.869105 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.869081    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a2f813d1-e334-46f2-adc5-d8d7d275d880" (UID: "a2f813d1-e334-46f2-adc5-d8d7d275d880"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:35:36.870937 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.870915    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst" (OuterVolumeSpecName: "kube-api-access-q6cst") pod "a2f813d1-e334-46f2-adc5-d8d7d275d880" (UID: "a2f813d1-e334-46f2-adc5-d8d7d275d880"). InnerVolumeSpecName "kube-api-access-q6cst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:35:36.970105 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.970082    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2f813d1-e334-46f2-adc5-d8d7d275d880-extensions-socket-volume\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:36.970105 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:36.970103    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cst\" (UniqueName: \"kubernetes.io/projected/a2f813d1-e334-46f2-adc5-d8d7d275d880-kube-api-access-q6cst\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:37.750183 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:37.750158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:37.752913 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:37.752891    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="a2f813d1-e334-46f2-adc5-d8d7d275d880" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:37.757859 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:37.757821    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cgt6c\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" podUID="a2f813d1-e334-46f2-adc5-d8d7d275d880" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cgt6c"
Jul 07 16:35:38.095495 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:38.095463    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f813d1-e334-46f2-adc5-d8d7d275d880" path="/var/lib/kubelet/pods/a2f813d1-e334-46f2-adc5-d8d7d275d880/volumes"
Jul 07 16:35:45.743532 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:45.743498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-phvsn"
Jul 07 16:35:45.744034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:45.743578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:45.744034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:45.743724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:35:45.845377 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:45.845351    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:45.845534 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:45.845511    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" podUID="9250921e-441b-458e-8289-315a544d54b6" containerName="manager" containerID="cri-o://b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9" gracePeriod=10
Jul 07 16:35:46.077864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.077840    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:46.133326 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.133300    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume\") pod \"9250921e-441b-458e-8289-315a544d54b6\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") "
Jul 07 16:35:46.133440 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.133331    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmb6g\" (UniqueName: \"kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g\") pod \"9250921e-441b-458e-8289-315a544d54b6\" (UID: \"9250921e-441b-458e-8289-315a544d54b6\") "
Jul 07 16:35:46.133691 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.133664    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9250921e-441b-458e-8289-315a544d54b6" (UID: "9250921e-441b-458e-8289-315a544d54b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:35:46.135306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.135282    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g" (OuterVolumeSpecName: "kube-api-access-mmb6g") pod "9250921e-441b-458e-8289-315a544d54b6" (UID: "9250921e-441b-458e-8289-315a544d54b6"). InnerVolumeSpecName "kube-api-access-mmb6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:35:46.234562 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.234538    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9250921e-441b-458e-8289-315a544d54b6-extensions-socket-volume\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:46.234562 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.234558    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmb6g\" (UniqueName: \"kubernetes.io/projected/9250921e-441b-458e-8289-315a544d54b6-kube-api-access-mmb6g\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:35:46.796409 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.796378    2565 generic.go:356] "Generic (PLEG): container finished" podID="9250921e-441b-458e-8289-315a544d54b6" containerID="b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9" exitCode=0
Jul 07 16:35:46.796853 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.796425    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"
Jul 07 16:35:46.796853 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.796464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" event={"ID":"9250921e-441b-458e-8289-315a544d54b6","Type":"ContainerDied","Data":"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"}
Jul 07 16:35:46.796853 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.796529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7" event={"ID":"9250921e-441b-458e-8289-315a544d54b6","Type":"ContainerDied","Data":"22f1eeac5ce208964f1bac29053fe2de615a405f0c4bd507356c62ab6e585875"}
Jul 07 16:35:46.796853 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.796562    2565 scope.go:117] "RemoveContainer" containerID="b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"
Jul 07 16:35:46.805123 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.805106    2565 scope.go:117] "RemoveContainer" containerID="b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"
Jul 07 16:35:46.805416 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:35:46.805399    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9\": container with ID starting with b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9 not found: ID does not exist" containerID="b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"
Jul 07 16:35:46.805466 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.805426    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9"} err="failed to get container status \"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9\": rpc error: code = NotFound desc = could not find container \"b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9\": container with ID starting with b82850cd1705e21ab05cbcb889fe69373e3325cad4efe29f11cda1c24bfe72e9 not found: ID does not exist"
Jul 07 16:35:46.844170 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.844149    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:46.849298 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:46.849277    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jn7g7"]
Jul 07 16:35:48.095548 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:35:48.095516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9250921e-441b-458e-8289-315a544d54b6" path="/var/lib/kubelet/pods/9250921e-441b-458e-8289-315a544d54b6/volumes"
Jul 07 16:36:02.347735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.347703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"]
Jul 07 16:36:02.348153 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.348061    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9250921e-441b-458e-8289-315a544d54b6" containerName="manager"
Jul 07 16:36:02.348153 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.348074    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9250921e-441b-458e-8289-315a544d54b6" containerName="manager"
Jul 07 16:36:02.348153 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.348143    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9250921e-441b-458e-8289-315a544d54b6" containerName="manager"
Jul 07 16:36:02.352975 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.352954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.356462 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.356443    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b47kb\""
Jul 07 16:36:02.372103 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.372082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"]
Jul 07 16:36:02.457818 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.457940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.457940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f63330-fb5b-4500-813e-978da828aa64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.457940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.457940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvmz\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-kube-api-access-fcvmz\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.458088 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.458088 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.457977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.458088 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.458056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f63330-fb5b-4500-813e-978da828aa64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.458088 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.458074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.558967 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.558936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f63330-fb5b-4500-813e-978da828aa64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.558967 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.558967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559134 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.558995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559134 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559239 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f63330-fb5b-4500-813e-978da828aa64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559239 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559239 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvmz\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-kube-api-access-fcvmz\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559378 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559339    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559389    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559621 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559599    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559653    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.559894 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.559870    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f63330-fb5b-4500-813e-978da828aa64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.561388 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.561361    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f63330-fb5b-4500-813e-978da828aa64-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.561630 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.561613    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f63330-fb5b-4500-813e-978da828aa64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.567600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.567579    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvmz\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-kube-api-access-fcvmz\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.567762 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.567744    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f63330-fb5b-4500-813e-978da828aa64-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wh6l4\" (UID: \"99f63330-fb5b-4500-813e-978da828aa64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.662513 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.662467    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:02.798661 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.798636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"]
Jul 07 16:36:02.801220 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:36:02.801198    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f63330_fb5b_4500_813e_978da828aa64.slice/crio-2d88f345d6f6a56aa4aca78f93f7565da58dfe01b4d7ecd5742e36ced3bfdf58 WatchSource:0}: Error finding container 2d88f345d6f6a56aa4aca78f93f7565da58dfe01b4d7ecd5742e36ced3bfdf58: Status 404 returned error can't find the container with id 2d88f345d6f6a56aa4aca78f93f7565da58dfe01b4d7ecd5742e36ced3bfdf58
Jul 07 16:36:02.803139 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.803109    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:36:02.803206 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.803173    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:36:02.803206 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.803202    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:36:02.880923 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.880897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" event={"ID":"99f63330-fb5b-4500-813e-978da828aa64","Type":"ContainerStarted","Data":"e16b31908655dae9f4c590dc293b204257e53bff699f97692baf2b8b0e9df88d"}
Jul 07 16:36:02.881048 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.880946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" event={"ID":"99f63330-fb5b-4500-813e-978da828aa64","Type":"ContainerStarted","Data":"2d88f345d6f6a56aa4aca78f93f7565da58dfe01b4d7ecd5742e36ced3bfdf58"}
Jul 07 16:36:02.921521 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:02.921433    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" podStartSLOduration=0.921415193 podStartE2EDuration="921.415193ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:02.921120908 +0000 UTC m=+761.454037572" watchObservedRunningTime="2026-07-07 16:36:02.921415193 +0000 UTC m=+761.454331837"
Jul 07 16:36:03.663379 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:03.663345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:03.669230 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:03.669200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:03.885236 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:03.885213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:03.885985 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:03.885970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:36:05.151884 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.151854    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:05.154421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.154400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:05.157034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.157013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r95wp\""
Jul 07 16:36:05.164870 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.164848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:05.278920 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.278899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4lz\" (UniqueName: \"kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz\") pod \"authorino-75d556bd77-t92zs\" (UID: \"b75b3c87-7bab-415e-8919-dae12da426c9\") " pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:05.380108 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.380085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4lz\" (UniqueName: \"kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz\") pod \"authorino-75d556bd77-t92zs\" (UID: \"b75b3c87-7bab-415e-8919-dae12da426c9\") " pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:05.391513 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.391495    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4lz\" (UniqueName: \"kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz\") pod \"authorino-75d556bd77-t92zs\" (UID: \"b75b3c87-7bab-415e-8919-dae12da426c9\") " pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:05.467993 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.467927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:05.618840 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.618818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:05.620471 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:36:05.620445    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb75b3c87_7bab_415e_8919_dae12da426c9.slice/crio-78622d04676d704d3b3c4b3f0091f11a6fc178308bc06747f7c67606e261c42a WatchSource:0}: Error finding container 78622d04676d704d3b3c4b3f0091f11a6fc178308bc06747f7c67606e261c42a: Status 404 returned error can't find the container with id 78622d04676d704d3b3c4b3f0091f11a6fc178308bc06747f7c67606e261c42a
Jul 07 16:36:05.894865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:05.894834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t92zs" event={"ID":"b75b3c87-7bab-415e-8919-dae12da426c9","Type":"ContainerStarted","Data":"78622d04676d704d3b3c4b3f0091f11a6fc178308bc06747f7c67606e261c42a"}
Jul 07 16:36:08.913514 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:08.913432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t92zs" event={"ID":"b75b3c87-7bab-415e-8919-dae12da426c9","Type":"ContainerStarted","Data":"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"}
Jul 07 16:36:08.935371 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:08.935321    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-t92zs" podStartSLOduration=0.911571084 podStartE2EDuration="3.935303828s" totalImagesPullingTime="3.023732744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:08.934260162 +0000 UTC m=+767.467176804" watchObservedRunningTime="2026-07-07 16:36:08.935303828 +0000 UTC m=+767.468220472"
Jul 07 16:36:40.806689 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:40.806647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:40.807261 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:40.806878    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-t92zs" podUID="b75b3c87-7bab-415e-8919-dae12da426c9" containerName="authorino" containerID="cri-o://c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5" gracePeriod=30
Jul 07 16:36:41.056619 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.056597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:41.068126 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.068100    2565 generic.go:356] "Generic (PLEG): container finished" podID="b75b3c87-7bab-415e-8919-dae12da426c9" containerID="c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5" exitCode=0
Jul 07 16:36:41.068126 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.068131    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t92zs"
Jul 07 16:36:41.068262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.068180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t92zs" event={"ID":"b75b3c87-7bab-415e-8919-dae12da426c9","Type":"ContainerDied","Data":"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"}
Jul 07 16:36:41.068262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.068220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t92zs" event={"ID":"b75b3c87-7bab-415e-8919-dae12da426c9","Type":"ContainerDied","Data":"78622d04676d704d3b3c4b3f0091f11a6fc178308bc06747f7c67606e261c42a"}
Jul 07 16:36:41.068262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.068243    2565 scope.go:117] "RemoveContainer" containerID="c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"
Jul 07 16:36:41.075868 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.075851    2565 scope.go:117] "RemoveContainer" containerID="c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"
Jul 07 16:36:41.076114 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:36:41.076096    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5\": container with ID starting with c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5 not found: ID does not exist" containerID="c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"
Jul 07 16:36:41.076173 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.076124    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5"} err="failed to get container status \"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5\": rpc error: code = NotFound desc = could not find container \"c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5\": container with ID starting with c5954d1eb7937ad2c6f78d754abed86e026c5c1552c1347e52a59c0f4cb9f7a5 not found: ID does not exist"
Jul 07 16:36:41.145714 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.145697    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4lz\" (UniqueName: \"kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz\") pod \"b75b3c87-7bab-415e-8919-dae12da426c9\" (UID: \"b75b3c87-7bab-415e-8919-dae12da426c9\") "
Jul 07 16:36:41.147571 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.147549    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz" (OuterVolumeSpecName: "kube-api-access-rz4lz") pod "b75b3c87-7bab-415e-8919-dae12da426c9" (UID: "b75b3c87-7bab-415e-8919-dae12da426c9"). InnerVolumeSpecName "kube-api-access-rz4lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:36:41.247302 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.247282    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4lz\" (UniqueName: \"kubernetes.io/projected/b75b3c87-7bab-415e-8919-dae12da426c9-kube-api-access-rz4lz\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:36:41.392763 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.392744    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:41.398998 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:41.398981    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t92zs"]
Jul 07 16:36:42.095097 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:36:42.095068    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75b3c87-7bab-415e-8919-dae12da426c9" path="/var/lib/kubelet/pods/b75b3c87-7bab-415e-8919-dae12da426c9/volumes"
Jul 07 16:37:19.669246 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.669219    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-667b548c96-c47mq"]
Jul 07 16:37:19.669701 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.669533    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75b3c87-7bab-415e-8919-dae12da426c9" containerName="authorino"
Jul 07 16:37:19.669701 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.669545    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75b3c87-7bab-415e-8919-dae12da426c9" containerName="authorino"
Jul 07 16:37:19.669701 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.669612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75b3c87-7bab-415e-8919-dae12da426c9" containerName="authorino"
Jul 07 16:37:19.671419 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.671404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.675523 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.675503    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 07 16:37:19.695502 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.695483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-667b548c96-c47mq"]
Jul 07 16:37:19.734902 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.734883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.734989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.734909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2vc\" (UniqueName: \"kubernetes.io/projected/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-kube-api-access-cp2vc\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.734989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.734929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-openshift-service-ca\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.835470 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.835443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.835589 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.835473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2vc\" (UniqueName: \"kubernetes.io/projected/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-kube-api-access-cp2vc\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.835589 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.835492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-openshift-service-ca\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.835589 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:37:19.835556    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 07 16:37:19.835744 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:37:19.835623    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls podName:f29c39ec-0805-4e71-8cab-d6fc364c3ac9 nodeName:}" failed. No retries permitted until 2026-07-07 16:37:20.335606527 +0000 UTC m=+838.868523148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls") pod "maas-api-667b548c96-c47mq" (UID: "f29c39ec-0805-4e71-8cab-d6fc364c3ac9") : secret "maas-api-serving-cert" not found
Jul 07 16:37:19.836128 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.836111    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-openshift-service-ca\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:19.854990 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:19.854965    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2vc\" (UniqueName: \"kubernetes.io/projected/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-kube-api-access-cp2vc\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:20.339951 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:20.339919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:20.342024 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:20.342003    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f29c39ec-0805-4e71-8cab-d6fc364c3ac9-maas-api-tls\") pod \"maas-api-667b548c96-c47mq\" (UID: \"f29c39ec-0805-4e71-8cab-d6fc364c3ac9\") " pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:20.581352 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:20.581318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:20.704726 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:20.704703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-667b548c96-c47mq"]
Jul 07 16:37:20.707283 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:37:20.707235    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29c39ec_0805_4e71_8cab_d6fc364c3ac9.slice/crio-a78731c2c49a116ba2ddaf56fba9cf20ab76f5afdd1f8f6806e6b1168738eb81 WatchSource:0}: Error finding container a78731c2c49a116ba2ddaf56fba9cf20ab76f5afdd1f8f6806e6b1168738eb81: Status 404 returned error can't find the container with id a78731c2c49a116ba2ddaf56fba9cf20ab76f5afdd1f8f6806e6b1168738eb81
Jul 07 16:37:21.254429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:21.254396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-667b548c96-c47mq" event={"ID":"f29c39ec-0805-4e71-8cab-d6fc364c3ac9","Type":"ContainerStarted","Data":"a78731c2c49a116ba2ddaf56fba9cf20ab76f5afdd1f8f6806e6b1168738eb81"}
Jul 07 16:37:23.267670 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:23.267633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-667b548c96-c47mq" event={"ID":"f29c39ec-0805-4e71-8cab-d6fc364c3ac9","Type":"ContainerStarted","Data":"e907b30a0a4f379b13be563489a39a86f753bbf69b8f1341059bce021aede58f"}
Jul 07 16:37:23.268101 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:23.267770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:23.294328 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:23.294277    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-667b548c96-c47mq" podStartSLOduration=2.204466684 podStartE2EDuration="4.29425772s" totalImagesPullingTime="2.089791036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:37:23.292807348 +0000 UTC m=+841.825724011" watchObservedRunningTime="2026-07-07 16:37:23.29425772 +0000 UTC m=+841.827174365"
Jul 07 16:37:28.291279 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.291248    2565 generic.go:356] "Generic (PLEG): container finished" podID="99f63330-fb5b-4500-813e-978da828aa64" containerID="e16b31908655dae9f4c590dc293b204257e53bff699f97692baf2b8b0e9df88d" exitCode=0
Jul 07 16:37:28.291645 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.291325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" event={"ID":"99f63330-fb5b-4500-813e-978da828aa64","Type":"ContainerDied","Data":"e16b31908655dae9f4c590dc293b204257e53bff699f97692baf2b8b0e9df88d"}
Jul 07 16:37:28.291715 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.291696    2565 scope.go:117] "RemoveContainer" containerID="e16b31908655dae9f4c590dc293b204257e53bff699f97692baf2b8b0e9df88d"
Jul 07 16:37:28.292759 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.292728    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:37:28.292841 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.292825    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:37:28.292886 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:28.292870    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:37:29.277574 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:29.277552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-667b548c96-c47mq"
Jul 07 16:37:29.298141 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:29.298102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" event={"ID":"99f63330-fb5b-4500-813e-978da828aa64","Type":"ContainerStarted","Data":"f3254288b5bb807f920ee48fcd707bf51f128bd378525b35f2370776958cf3dc"}
Jul 07 16:37:29.662952 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:29.662913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:37:29.664075 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:29.664047    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wh6l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 07 16:37:29.664211 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:29.664095    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" podUID="99f63330-fb5b-4500-813e-978da828aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 07 16:37:30.662938 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:30.662906    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wh6l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 07 16:37:30.663309 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:30.662957    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" podUID="99f63330-fb5b-4500-813e-978da828aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 07 16:37:31.663007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:31.662976    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wh6l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 07 16:37:31.663354 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:31.663025    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" podUID="99f63330-fb5b-4500-813e-978da828aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 07 16:37:32.663822 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:32.663762    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wh6l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 07 16:37:32.664179 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:32.663840    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4" podUID="99f63330-fb5b-4500-813e-978da828aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 07 16:37:33.668767 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:33.668735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:37:33.669221 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:33.669128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:37:33.669897 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:33.669876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wh6l4"
Jul 07 16:37:44.896162 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.896132    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"]
Jul 07 16:37:44.898992 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.898974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.907039 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.907018    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 07 16:37:44.908260 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.908245    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 16:37:44.917198 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.917177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 16:37:44.917596 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.917579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwbt\""
Jul 07 16:37:44.919578 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.919560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"]
Jul 07 16:37:44.923992 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.923969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.924100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.924008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9lh\" (UniqueName: \"kubernetes.io/projected/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-kube-api-access-qz9lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.924100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.924055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.924212 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.924153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.924212 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.924183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:44.924311 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:44.924270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025474 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9lh\" (UniqueName: \"kubernetes.io/projected/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-kube-api-access-qz9lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025567 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.025949 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025931    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.026007 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.025977    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.026048 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.026006    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.027588 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.027571    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.027748 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.027731    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.036109 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.036090    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9lh\" (UniqueName: \"kubernetes.io/projected/ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068-kube-api-access-qz9lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-knk4g\" (UID: \"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.208634 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.208571    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:45.337649 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.337619    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"]
Jul 07 16:37:45.339493 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:37:45.339466    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab86ff7e_6a1c_48a9_b7ee_c1a5e6f24068.slice/crio-c85804d9f64ebebed5fe34d6a6ca74855afa2fa25cb417b65792e5db44b4c7d4 WatchSource:0}: Error finding container c85804d9f64ebebed5fe34d6a6ca74855afa2fa25cb417b65792e5db44b4c7d4: Status 404 returned error can't find the container with id c85804d9f64ebebed5fe34d6a6ca74855afa2fa25cb417b65792e5db44b4c7d4
Jul 07 16:37:45.371481 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:45.371455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g" event={"ID":"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068","Type":"ContainerStarted","Data":"c85804d9f64ebebed5fe34d6a6ca74855afa2fa25cb417b65792e5db44b4c7d4"}
Jul 07 16:37:46.378062 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:46.378036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g" event={"ID":"ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068","Type":"ContainerStarted","Data":"85ef00982ef1c8a6ad17192819797b6c6df9bed2de7d4cd39b7462c9161c38e5"}
Jul 07 16:37:46.401890 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:46.401840    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g" podStartSLOduration=1.449248181 podStartE2EDuration="2.401777038s" totalImagesPullingTime="952.528857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:37:46.401325951 +0000 UTC m=+864.934242685" watchObservedRunningTime="2026-07-07 16:37:46.401777038 +0000 UTC m=+864.934693681"
Jul 07 16:37:55.209409 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:55.209372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:55.209976 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:55.209469    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:55.221401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:55.221372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:37:55.430099 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:37:55.430078    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-knk4g"
Jul 07 16:38:04.073372 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.073345    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"]
Jul 07 16:38:04.076057 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.076036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.078777 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.078754    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 07 16:38:04.096743 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.096720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"]
Jul 07 16:38:04.161313 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.161425 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvt7\" (UniqueName: \"kubernetes.io/projected/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-kube-api-access-wlvt7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.161425 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.161425 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.161520 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.161520 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.161490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262105 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvt7\" (UniqueName: \"kubernetes.io/projected/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-kube-api-access-wlvt7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262377 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262529 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262549    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.262652 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.262638    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.264386 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.264366    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.264548 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.264533    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.272583 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.272562    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvt7\" (UniqueName: \"kubernetes.io/projected/8c4b8acb-f560-4b87-bac3-41c8baf74f7f-kube-api-access-wlvt7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc\" (UID: \"8c4b8acb-f560-4b87-bac3-41c8baf74f7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.386549 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.386480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:04.517317 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:04.517295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"]
Jul 07 16:38:04.519292 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:38:04.519267    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4b8acb_f560_4b87_bac3_41c8baf74f7f.slice/crio-c073b5d824b4107c4c11499a0636a3e6f33596fb2f622c7fb49fe4224ea511e1 WatchSource:0}: Error finding container c073b5d824b4107c4c11499a0636a3e6f33596fb2f622c7fb49fe4224ea511e1: Status 404 returned error can't find the container with id c073b5d824b4107c4c11499a0636a3e6f33596fb2f622c7fb49fe4224ea511e1
Jul 07 16:38:05.465887 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:05.465843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc" event={"ID":"8c4b8acb-f560-4b87-bac3-41c8baf74f7f","Type":"ContainerStarted","Data":"6a3de2347724240006a6bf9cea761f7efec837ef964f66fd10ddd3b8d9bf46e0"}
Jul 07 16:38:05.466253 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:05.465937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc" event={"ID":"8c4b8acb-f560-4b87-bac3-41c8baf74f7f","Type":"ContainerStarted","Data":"c073b5d824b4107c4c11499a0636a3e6f33596fb2f622c7fb49fe4224ea511e1"}
Jul 07 16:38:05.488235 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:05.488165    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc" podStartSLOduration=1.318365123 podStartE2EDuration="1.488145145s" totalImagesPullingTime="169.780022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:38:05.48547358 +0000 UTC m=+884.018390246" watchObservedRunningTime="2026-07-07 16:38:05.488145145 +0000 UTC m=+884.021061788"
Jul 07 16:38:14.387360 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:14.387290    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:14.387360 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:14.387360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:14.398798 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:14.398767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:14.520176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:14.520152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc"
Jul 07 16:38:16.994612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:16.994576    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"]
Jul 07 16:38:16.997564 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:16.997542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.001027 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.001008    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 16:38:17.019161 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.019135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"]
Jul 07 16:38:17.055769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.055895 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.055895 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.055895 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btg9m\" (UniqueName: \"kubernetes.io/projected/9a04425b-b330-4c69-9e1f-f61657c72ed8-kube-api-access-btg9m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.055895 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.056019 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.055954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a04425b-b330-4c69-9e1f-f61657c72ed8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157241 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157241 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btg9m\" (UniqueName: \"kubernetes.io/projected/9a04425b-b330-4c69-9e1f-f61657c72ed8-kube-api-access-btg9m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157341 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157341 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a04425b-b330-4c69-9e1f-f61657c72ed8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157448 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157500 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157471    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157719 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157700    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.157905 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.157769    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.159902 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.159869    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a04425b-b330-4c69-9e1f-f61657c72ed8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.160313 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.160294    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a04425b-b330-4c69-9e1f-f61657c72ed8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.172536 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.172515    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btg9m\" (UniqueName: \"kubernetes.io/projected/9a04425b-b330-4c69-9e1f-f61657c72ed8-kube-api-access-btg9m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf\" (UID: \"9a04425b-b330-4c69-9e1f-f61657c72ed8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.306517 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.306466    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:17.440964 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.440935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"]
Jul 07 16:38:17.444043 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:38:17.444018    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a04425b_b330_4c69_9e1f_f61657c72ed8.slice/crio-dc3d0abd5a81f3ba99c6cb9e2e93ee8a185daf06360688986564d7778cf6d0a4 WatchSource:0}: Error finding container dc3d0abd5a81f3ba99c6cb9e2e93ee8a185daf06360688986564d7778cf6d0a4: Status 404 returned error can't find the container with id dc3d0abd5a81f3ba99c6cb9e2e93ee8a185daf06360688986564d7778cf6d0a4
Jul 07 16:38:17.524767 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:17.524732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf" event={"ID":"9a04425b-b330-4c69-9e1f-f61657c72ed8","Type":"ContainerStarted","Data":"dc3d0abd5a81f3ba99c6cb9e2e93ee8a185daf06360688986564d7778cf6d0a4"}
Jul 07 16:38:18.530886 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:18.530853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf" event={"ID":"9a04425b-b330-4c69-9e1f-f61657c72ed8","Type":"ContainerStarted","Data":"328a00b9bfb00a46924550e6cf51fdd5b3210aa4093e66379a49adf510a6432a"}
Jul 07 16:38:18.563432 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:18.563382    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf" podStartSLOduration=2.4013480830000002 podStartE2EDuration="2.563366757s" totalImagesPullingTime="162.018674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:38:18.562940754 +0000 UTC m=+897.095857397" watchObservedRunningTime="2026-07-07 16:38:18.563366757 +0000 UTC m=+897.096283399"
Jul 07 16:38:27.307237 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:27.307202    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:27.307675 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:27.307258    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:27.319696 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:27.319672    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:38:27.580845 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:38:27.580824    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf"
Jul 07 16:45:28.931845 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:28.931813    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:45:28.932243 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:28.931997    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" podUID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" containerName="manager" containerID="cri-o://f694ab38a1a655b171385b8e718653645a12b315c92784c0bd826da2fec504cd" gracePeriod=10
Jul 07 16:45:31.581037 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.581002    2565 generic.go:356] "Generic (PLEG): container finished" podID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" containerID="f694ab38a1a655b171385b8e718653645a12b315c92784c0bd826da2fec504cd" exitCode=0
Jul 07 16:45:31.581339 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.581056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" event={"ID":"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a","Type":"ContainerDied","Data":"f694ab38a1a655b171385b8e718653645a12b315c92784c0bd826da2fec504cd"}
Jul 07 16:45:31.679822 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.679773    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:45:31.705973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.705950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume\") pod \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") "
Jul 07 16:45:31.706104 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.706085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqvjr\" (UniqueName: \"kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr\") pod \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\" (UID: \"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a\") "
Jul 07 16:45:31.706374 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.706344    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" (UID: "bd4a8b98-ff21-4ab4-a723-3f77d8750b7a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:45:31.708118 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.708088    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr" (OuterVolumeSpecName: "kube-api-access-cqvjr") pod "bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" (UID: "bd4a8b98-ff21-4ab4-a723-3f77d8750b7a"). InnerVolumeSpecName "kube-api-access-cqvjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:45:31.807238 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.807182    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvjr\" (UniqueName: \"kubernetes.io/projected/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-kube-api-access-cqvjr\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:45:31.807238 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:31.807203    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a-extensions-socket-volume\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:45:32.587231 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:32.587184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt" event={"ID":"bd4a8b98-ff21-4ab4-a723-3f77d8750b7a","Type":"ContainerDied","Data":"2d54a44fc9eb504fd6cdc571f7455126ab45bd25f28e8afff84a16fb69e8c174"}
Jul 07 16:45:32.587231 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:32.587207    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"
Jul 07 16:45:32.587681 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:32.587258    2565 scope.go:117] "RemoveContainer" containerID="f694ab38a1a655b171385b8e718653645a12b315c92784c0bd826da2fec504cd"
Jul 07 16:45:32.611261 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:32.611230    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:45:32.617925 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:32.617900    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p9pnt"]
Jul 07 16:45:34.095605 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:45:34.095557    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" path="/var/lib/kubelet/pods/bd4a8b98-ff21-4ab4-a723-3f77d8750b7a/volumes"
Jul 07 16:46:30.289643 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.289616    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"]
Jul 07 16:46:30.290045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.289963    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" containerName="manager"
Jul 07 16:46:30.290045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.289975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" containerName="manager"
Jul 07 16:46:30.290045 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.290037    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4a8b98-ff21-4ab4-a723-3f77d8750b7a" containerName="manager"
Jul 07 16:46:30.294308 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.294292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.297941 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.297922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gpg2n\""
Jul 07 16:46:30.307004 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.306977    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"]
Jul 07 16:46:30.374438 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.374417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22421eb8-6052-41ca-b805-57d81ff82bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.374540 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.374469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zd9\" (UniqueName: \"kubernetes.io/projected/22421eb8-6052-41ca-b805-57d81ff82bb8-kube-api-access-f6zd9\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.475730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.475691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22421eb8-6052-41ca-b805-57d81ff82bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.475864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.475773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zd9\" (UniqueName: \"kubernetes.io/projected/22421eb8-6052-41ca-b805-57d81ff82bb8-kube-api-access-f6zd9\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.476095 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.476075    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22421eb8-6052-41ca-b805-57d81ff82bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.490375 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.490340    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zd9\" (UniqueName: \"kubernetes.io/projected/22421eb8-6052-41ca-b805-57d81ff82bb8-kube-api-access-f6zd9\") pod \"kuadrant-operator-controller-manager-86878c769f-lckwd\" (UID: \"22421eb8-6052-41ca-b805-57d81ff82bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.604300 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.604280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.772695 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.772662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"]
Jul 07 16:46:30.775399 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:46:30.775374    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22421eb8_6052_41ca_b805_57d81ff82bb8.slice/crio-8c51c3f4b1a5b0b6a96d2f0bba732ee363596dc3c12d45ecb6cda0bdb875d639 WatchSource:0}: Error finding container 8c51c3f4b1a5b0b6a96d2f0bba732ee363596dc3c12d45ecb6cda0bdb875d639: Status 404 returned error can't find the container with id 8c51c3f4b1a5b0b6a96d2f0bba732ee363596dc3c12d45ecb6cda0bdb875d639
Jul 07 16:46:30.897550 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.897497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd" event={"ID":"22421eb8-6052-41ca-b805-57d81ff82bb8","Type":"ContainerStarted","Data":"d91c45e272249d7fce1bd203f11e8eb98fb7104194cc663c10db81fd209a50fa"}
Jul 07 16:46:30.897632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.897558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd" event={"ID":"22421eb8-6052-41ca-b805-57d81ff82bb8","Type":"ContainerStarted","Data":"8c51c3f4b1a5b0b6a96d2f0bba732ee363596dc3c12d45ecb6cda0bdb875d639"}
Jul 07 16:46:30.897632 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.897582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:46:30.965962 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:30.965922    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd" podStartSLOduration=0.965907157 podStartE2EDuration="965.907157ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:30.964686131 +0000 UTC m=+1389.497602774" watchObservedRunningTime="2026-07-07 16:46:30.965907157 +0000 UTC m=+1389.498823801"
Jul 07 16:46:41.904770 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:46:41.904726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lckwd"
Jul 07 16:53:19.914403 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:19.914370    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:19.917771 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:19.917754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:19.925474 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:19.925455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6306fce7-openshift-default-dockercfg-hszh5\""
Jul 07 16:53:19.972023 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:19.971988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:20.019355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019355 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf8s\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019843 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.019843 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.019634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120152 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120291 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120291 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120399 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120399 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120515 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120515 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf8s\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120515 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120515 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120732 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120608    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120732 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120683    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.120851 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.120798    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.121054 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.121031    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.121107 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.121062    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.122135 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.122116    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.122406 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.122391    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.144046 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.144003    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf8s\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.144111 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.144075    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token\") pod \"e2e-ait-6306fce7-openshift-default-777498cd7-7nfff\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") " pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.228097 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.228075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:20.363977 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.363950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:20.365132 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:53:20.365108    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8a778e_f48c_4bb8_884c_c3ddd6e37c73.slice/crio-c26b4ee4f4792e9d2f24998615264f6e1aa3c3ceadf94af5a63cf5178350b1c2 WatchSource:0}: Error finding container c26b4ee4f4792e9d2f24998615264f6e1aa3c3ceadf94af5a63cf5178350b1c2: Status 404 returned error can't find the container with id c26b4ee4f4792e9d2f24998615264f6e1aa3c3ceadf94af5a63cf5178350b1c2
Jul 07 16:53:20.367082 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.367046    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:20.367154 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.367142    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:20.367195 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.367171    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:20.819677 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.819637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" event={"ID":"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73","Type":"ContainerStarted","Data":"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"}
Jul 07 16:53:20.819891 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.819697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" event={"ID":"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73","Type":"ContainerStarted","Data":"c26b4ee4f4792e9d2f24998615264f6e1aa3c3ceadf94af5a63cf5178350b1c2"}
Jul 07 16:53:20.848501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:20.848454    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" podStartSLOduration=1.848439406 podStartE2EDuration="1.848439406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:20.84547647 +0000 UTC m=+1799.378393136" watchObservedRunningTime="2026-07-07 16:53:20.848439406 +0000 UTC m=+1799.381356049"
Jul 07 16:53:21.229172 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:21.229103    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:21.233189 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:21.233168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:21.824305 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:21.824272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:21.825112 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:21.825092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:29.478380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:29.478344    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:29.478734 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:29.478563    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" podUID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" containerName="istio-proxy" containerID="cri-o://89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14" gracePeriod=30
Jul 07 16:53:34.428012 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.427989    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:34.564564 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjf8s\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564607    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564764    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy\") pod \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\" (UID: \"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73\") "
Jul 07 16:53:34.564956 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.564817    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:34.565159 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.565059    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.565209 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.565188    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:34.565258 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.565201    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:34.565258 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.565120    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:53:34.565587 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.565560    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:34.566780 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.566751    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:53:34.566780 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.566815    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:34.566946 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.566839    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s" (OuterVolumeSpecName: "kube-api-access-vjf8s") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "kube-api-access-vjf8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:34.566946 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.566921    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" (UID: "9e8a778e-f48c-4bb8-884c-c3ddd6e37c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:34.666363 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666338    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666363 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666361    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666373    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666384    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666394    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666405    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666416    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjf8s\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-kube-api-access-vjf8s\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.666496 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.666428    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:34.887144 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.887077    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" containerID="89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14" exitCode=0
Jul 07 16:53:34.887144 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.887132    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"
Jul 07 16:53:34.887270 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.887162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" event={"ID":"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73","Type":"ContainerDied","Data":"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"}
Jul 07 16:53:34.887270 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.887249    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff" event={"ID":"9e8a778e-f48c-4bb8-884c-c3ddd6e37c73","Type":"ContainerDied","Data":"c26b4ee4f4792e9d2f24998615264f6e1aa3c3ceadf94af5a63cf5178350b1c2"}
Jul 07 16:53:34.887343 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.887278    2565 scope.go:117] "RemoveContainer" containerID="89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"
Jul 07 16:53:34.896101 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.896086    2565 scope.go:117] "RemoveContainer" containerID="89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"
Jul 07 16:53:34.896347 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:34.896332    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14\": container with ID starting with 89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14 not found: ID does not exist" containerID="89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"
Jul 07 16:53:34.896409 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.896353    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14"} err="failed to get container status \"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14\": rpc error: code = NotFound desc = could not find container \"89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14\": container with ID starting with 89fc8501eb07a0597b3192258642e7bf17cc8e0a59b13ef56553e2e454259e14 not found: ID does not exist"
Jul 07 16:53:34.913890 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.913868    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:34.920189 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:34.920170    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6306fce7-openshift-default-777498cd7-7nfff"]
Jul 07 16:53:36.095046 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.095013    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" path="/var/lib/kubelet/pods/9e8a778e-f48c-4bb8-884c-c3ddd6e37c73/volumes"
Jul 07 16:53:36.476646 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.476580    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:36.476958 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.476945    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" containerName="istio-proxy"
Jul 07 16:53:36.477004 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.476959    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" containerName="istio-proxy"
Jul 07 16:53:36.477037 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.477019    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8a778e-f48c-4bb8-884c-c3ddd6e37c73" containerName="istio-proxy"
Jul 07 16:53:36.479659 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.479637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.482837 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.482821    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-26b38dd4-openshift-default-dockercfg-q4t7m\""
Jul 07 16:53:36.502728 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.502705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:36.579769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.579905 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.579905 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.579905 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.580041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.580041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.580041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.579966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l8j\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.580041 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.580010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.580192 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.580056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680754 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680754 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l8j\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.680953 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681242 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.680957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681242 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.681191    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681336 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.681252    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681336 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.681327    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681663 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.681636    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.681737 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.681720    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.683051 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.683032    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.683190 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.683174    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.693653 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.693634    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l8j\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.695501 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.695480    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token\") pod \"e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") " pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.790016 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.789960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:36.926458 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.926433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:36.928344 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:53:36.928321    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15252fa2_0f7c_4beb_9682_b0d69e61d912.slice/crio-53f7fe56db84b232b6a66d85d5119332d4e906f6eb613999150ee6ad97aa3990 WatchSource:0}: Error finding container 53f7fe56db84b232b6a66d85d5119332d4e906f6eb613999150ee6ad97aa3990: Status 404 returned error can't find the container with id 53f7fe56db84b232b6a66d85d5119332d4e906f6eb613999150ee6ad97aa3990
Jul 07 16:53:36.930219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.930187    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:36.930295 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.930250    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:36.930295 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:36.930277    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:53:37.823439 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.823410    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:37.825909 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.825888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.834812 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.834717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\""
Jul 07 16:53:37.835112 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.835078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-26b38dd4-serving-cert\""
Jul 07 16:53:37.854678 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.854656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:37.891288 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.891267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.891378 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.891299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcvc\" (UniqueName: \"kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.891378 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.891370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.906134 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.906109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" event={"ID":"15252fa2-0f7c-4beb-9682-b0d69e61d912","Type":"ContainerStarted","Data":"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"}
Jul 07 16:53:37.906220 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.906153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" event={"ID":"15252fa2-0f7c-4beb-9682-b0d69e61d912","Type":"ContainerStarted","Data":"53f7fe56db84b232b6a66d85d5119332d4e906f6eb613999150ee6ad97aa3990"}
Jul 07 16:53:37.933161 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.933121    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" podStartSLOduration=1.933109935 podStartE2EDuration="1.933109935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:37.932770793 +0000 UTC m=+1816.465687437" watchObservedRunningTime="2026-07-07 16:53:37.933109935 +0000 UTC m=+1816.466026577"
Jul 07 16:53:37.991761 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.991739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.991880 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.991770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcvc\" (UniqueName: \"kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.991935 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:37.991883    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-26b38dd4-serving-cert: secret "maas-api-e2e-ait-26b38dd4-serving-cert" not found
Jul 07 16:53:37.991970 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:37.991946    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls podName:6b8ebc46-973e-4c2d-af00-cb91751309bf nodeName:}" failed. No retries permitted until 2026-07-07 16:53:38.491931798 +0000 UTC m=+1817.024848421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls") pod "maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" (UID: "6b8ebc46-973e-4c2d-af00-cb91751309bf") : secret "maas-api-e2e-ait-26b38dd4-serving-cert" not found
Jul 07 16:53:37.992053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.992035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:37.992639 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:37.992622    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:38.005153 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.005130    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcvc\" (UniqueName: \"kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:38.496258 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.496227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:38.498743 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.498722    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") pod \"maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") " pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:38.739437 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.739403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:38.790455 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.790373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:38.797341 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.797318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:38.880515 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.880489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:38.882252 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:53:38.882228    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8ebc46_973e_4c2d_af00_cb91751309bf.slice/crio-65575110fed725bc43c71a51eb797e9b3f4cd5869771c7954026f6bd81b7674c WatchSource:0}: Error finding container 65575110fed725bc43c71a51eb797e9b3f4cd5869771c7954026f6bd81b7674c: Status 404 returned error can't find the container with id 65575110fed725bc43c71a51eb797e9b3f4cd5869771c7954026f6bd81b7674c
Jul 07 16:53:38.884036 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.884018    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:53:38.912464 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.912433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" event={"ID":"6b8ebc46-973e-4c2d-af00-cb91751309bf","Type":"ContainerStarted","Data":"65575110fed725bc43c71a51eb797e9b3f4cd5869771c7954026f6bd81b7674c"}
Jul 07 16:53:38.912762 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.912737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:38.913810 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:38.913764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:40.923801 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:40.923739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" event={"ID":"6b8ebc46-973e-4c2d-af00-cb91751309bf","Type":"ContainerStarted","Data":"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"}
Jul 07 16:53:40.924150 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:40.923931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:40.953018 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:40.952972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" podStartSLOduration=2.835847425 podStartE2EDuration="3.952958371s" totalImagesPullingTime="1.117110946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:40.950729632 +0000 UTC m=+1819.483646275" watchObservedRunningTime="2026-07-07 16:53:40.952958371 +0000 UTC m=+1819.485875013"
Jul 07 16:53:45.926809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:45.925099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:45.926809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:45.925340    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" podUID="6b8ebc46-973e-4c2d-af00-cb91751309bf" containerName="maas-api" containerID="cri-o://a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f" gracePeriod=30
Jul 07 16:53:45.940310 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:45.940285    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:46.187978 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.187929    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:46.356809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.356761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfcvc\" (UniqueName: \"kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc\") pod \"6b8ebc46-973e-4c2d-af00-cb91751309bf\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") "
Jul 07 16:53:46.356998 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.356880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca\") pod \"6b8ebc46-973e-4c2d-af00-cb91751309bf\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") "
Jul 07 16:53:46.356998 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.356913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") pod \"6b8ebc46-973e-4c2d-af00-cb91751309bf\" (UID: \"6b8ebc46-973e-4c2d-af00-cb91751309bf\") "
Jul 07 16:53:46.357215 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.357185    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b8ebc46-973e-4c2d-af00-cb91751309bf" (UID: "6b8ebc46-973e-4c2d-af00-cb91751309bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:53:46.358838 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.358815    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b8ebc46-973e-4c2d-af00-cb91751309bf" (UID: "6b8ebc46-973e-4c2d-af00-cb91751309bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:53:46.358948 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.358931    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc" (OuterVolumeSpecName: "kube-api-access-hfcvc") pod "6b8ebc46-973e-4c2d-af00-cb91751309bf" (UID: "6b8ebc46-973e-4c2d-af00-cb91751309bf"). InnerVolumeSpecName "kube-api-access-hfcvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:46.458601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.458536    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcvc\" (UniqueName: \"kubernetes.io/projected/6b8ebc46-973e-4c2d-af00-cb91751309bf-kube-api-access-hfcvc\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:46.458601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.458561    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ebc46-973e-4c2d-af00-cb91751309bf-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:46.458601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.458570    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b8ebc46-973e-4c2d-af00-cb91751309bf-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:46.955419 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.955385    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b8ebc46-973e-4c2d-af00-cb91751309bf" containerID="a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f" exitCode=0
Jul 07 16:53:46.955864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.955448    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"
Jul 07 16:53:46.955864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.955479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" event={"ID":"6b8ebc46-973e-4c2d-af00-cb91751309bf","Type":"ContainerDied","Data":"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"}
Jul 07 16:53:46.955864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.955543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6" event={"ID":"6b8ebc46-973e-4c2d-af00-cb91751309bf","Type":"ContainerDied","Data":"65575110fed725bc43c71a51eb797e9b3f4cd5869771c7954026f6bd81b7674c"}
Jul 07 16:53:46.955864 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.955578    2565 scope.go:117] "RemoveContainer" containerID="a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"
Jul 07 16:53:46.964455 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.964434    2565 scope.go:117] "RemoveContainer" containerID="a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"
Jul 07 16:53:46.964768 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:46.964744    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f\": container with ID starting with a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f not found: ID does not exist" containerID="a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"
Jul 07 16:53:46.964887 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.964774    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f"} err="failed to get container status \"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f\": rpc error: code = NotFound desc = could not find container \"a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f\": container with ID starting with a66d1def93f883627a624a1eec70cae795beea0eea93e8efffa1d6b32b67864f not found: ID does not exist"
Jul 07 16:53:46.983902 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.983886    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:46.985481 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:46.985460    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-26b38dd4-66f6d48cd4-47gv6"]
Jul 07 16:53:47.204694 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:47.204662    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:47.204911 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:47.204879    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" podUID="15252fa2-0f7c-4beb-9682-b0d69e61d912" containerName="istio-proxy" containerID="cri-o://0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb" gracePeriod=30
Jul 07 16:53:48.095762 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:48.095733    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b8ebc46-973e-4c2d-af00-cb91751309bf" path="/var/lib/kubelet/pods/6b8ebc46-973e-4c2d-af00-cb91751309bf/volumes"
Jul 07 16:53:51.359252 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.359204    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:51.500837 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.500810    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.500974 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.500857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.500974 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.500884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.500974 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.500918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501129 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501113    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:51.501186 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501279    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501293    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2l8j\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501340    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy\") pod \"15252fa2-0f7c-4beb-9682-b0d69e61d912\" (UID: \"15252fa2-0f7c-4beb-9682-b0d69e61d912\") "
Jul 07 16:53:51.501565 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501442    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:51.501940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501655    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15252fa2-0f7c-4beb-9682-b0d69e61d912-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.501940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501672    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.501940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501686    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.501940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501698    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.501940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.501721    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data" (OuterVolumeSpecName: "istio-data") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:51.503162 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.503138    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:53:51.504398 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.504367    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token" (OuterVolumeSpecName: "istio-token") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:51.504484 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.504379    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:53:51.504484 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.504410    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j" (OuterVolumeSpecName: "kube-api-access-l2l8j") pod "15252fa2-0f7c-4beb-9682-b0d69e61d912" (UID: "15252fa2-0f7c-4beb-9682-b0d69e61d912"). InnerVolumeSpecName "kube-api-access-l2l8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:51.602908 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.602877    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2l8j\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-kube-api-access-l2l8j\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.602908 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.602902    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.602908 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.602911    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.603106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.602919    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.603106 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.602926    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15252fa2-0f7c-4beb-9682-b0d69e61d912-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:53:51.981259 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.981226    2565 generic.go:356] "Generic (PLEG): container finished" podID="15252fa2-0f7c-4beb-9682-b0d69e61d912" containerID="0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb" exitCode=0
Jul 07 16:53:51.981401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.981274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"
Jul 07 16:53:51.981401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.981288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" event={"ID":"15252fa2-0f7c-4beb-9682-b0d69e61d912","Type":"ContainerDied","Data":"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"}
Jul 07 16:53:51.981401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.981328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp" event={"ID":"15252fa2-0f7c-4beb-9682-b0d69e61d912","Type":"ContainerDied","Data":"53f7fe56db84b232b6a66d85d5119332d4e906f6eb613999150ee6ad97aa3990"}
Jul 07 16:53:51.981401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.981349    2565 scope.go:117] "RemoveContainer" containerID="0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"
Jul 07 16:53:51.990855 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.990838    2565 scope.go:117] "RemoveContainer" containerID="0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"
Jul 07 16:53:51.991119 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:51.991103    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb\": container with ID starting with 0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb not found: ID does not exist" containerID="0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"
Jul 07 16:53:51.991164 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:51.991125    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb"} err="failed to get container status \"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb\": rpc error: code = NotFound desc = could not find container \"0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb\": container with ID starting with 0ea1acb5fa8caa88a4d34b207b616469f5032fef7c4478044a5f49d9905832fb not found: ID does not exist"
Jul 07 16:53:52.004401 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:52.004376    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:52.010490 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:52.010471    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-26b38dd4-openshift-default-6769c8df-d8rfp"]
Jul 07 16:53:52.095272 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:52.095250    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15252fa2-0f7c-4beb-9682-b0d69e61d912" path="/var/lib/kubelet/pods/15252fa2-0f7c-4beb-9682-b0d69e61d912/volumes"
Jul 07 16:53:54.858504 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858472    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:53:54.858865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858837    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15252fa2-0f7c-4beb-9682-b0d69e61d912" containerName="istio-proxy"
Jul 07 16:53:54.858865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858847    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="15252fa2-0f7c-4beb-9682-b0d69e61d912" containerName="istio-proxy"
Jul 07 16:53:54.858865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858854    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8ebc46-973e-4c2d-af00-cb91751309bf" containerName="maas-api"
Jul 07 16:53:54.858865 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858860    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8ebc46-973e-4c2d-af00-cb91751309bf" containerName="maas-api"
Jul 07 16:53:54.858989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858920    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="15252fa2-0f7c-4beb-9682-b0d69e61d912" containerName="istio-proxy"
Jul 07 16:53:54.858989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.858927    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8ebc46-973e-4c2d-af00-cb91751309bf" containerName="maas-api"
Jul 07 16:53:54.860813 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.860797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:54.863544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.863517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-f2f03512-serving-cert\""
Jul 07 16:53:54.863544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.863535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\""
Jul 07 16:53:54.878882 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:54.878862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:53:55.027945 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.027914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdgn\" (UniqueName: \"kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.028096 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.027980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.028096 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.028016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.129009 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.128949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdgn\" (UniqueName: \"kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.129009 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.129001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.129140 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.129031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.129175 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:55.129152    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-f2f03512-serving-cert: secret "maas-api-e2e-derive-f2f03512-serving-cert" not found
Jul 07 16:53:55.129236 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:53:55.129224    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls podName:66847664-78a5-40cf-90fd-aeaa9cd993d7 nodeName:}" failed. No retries permitted until 2026-07-07 16:53:55.629203597 +0000 UTC m=+1834.162120229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls") pod "maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" (UID: "66847664-78a5-40cf-90fd-aeaa9cd993d7") : secret "maas-api-e2e-derive-f2f03512-serving-cert" not found
Jul 07 16:53:55.129664 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.129647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.149068 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.149041    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdgn\" (UniqueName: \"kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.632422 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.632392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.634594 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.634574    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") pod \"maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") " pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.770648 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.770617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:55.912534 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:55.912433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:53:55.914868 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:53:55.914837    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66847664_78a5_40cf_90fd_aeaa9cd993d7.slice/crio-daec79334256bb646ee436c900610cc757625f9f52264ebceed04d78987fa8a4 WatchSource:0}: Error finding container daec79334256bb646ee436c900610cc757625f9f52264ebceed04d78987fa8a4: Status 404 returned error can't find the container with id daec79334256bb646ee436c900610cc757625f9f52264ebceed04d78987fa8a4
Jul 07 16:53:56.004735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:56.004704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" event={"ID":"66847664-78a5-40cf-90fd-aeaa9cd993d7","Type":"ContainerStarted","Data":"daec79334256bb646ee436c900610cc757625f9f52264ebceed04d78987fa8a4"}
Jul 07 16:53:57.012114 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:57.012077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" event={"ID":"66847664-78a5-40cf-90fd-aeaa9cd993d7","Type":"ContainerStarted","Data":"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"}
Jul 07 16:53:57.012475 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:57.012243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:53:57.033346 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:53:57.033300    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" podStartSLOduration=2.499229797 podStartE2EDuration="3.033283653s" totalImagesPullingTime="534.053856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:57.032310724 +0000 UTC m=+1835.565227366" watchObservedRunningTime="2026-07-07 16:53:57.033283653 +0000 UTC m=+1835.566200295"
Jul 07 16:54:02.223504 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.223474    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:54:02.223912 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.223634    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" podUID="66847664-78a5-40cf-90fd-aeaa9cd993d7" containerName="maas-api" containerID="cri-o://5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e" gracePeriod=30
Jul 07 16:54:02.228361 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.228342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:54:02.465816 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.465774    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:54:02.585231 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.585202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") pod \"66847664-78a5-40cf-90fd-aeaa9cd993d7\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") "
Jul 07 16:54:02.585359 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.585257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxdgn\" (UniqueName: \"kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn\") pod \"66847664-78a5-40cf-90fd-aeaa9cd993d7\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") "
Jul 07 16:54:02.585422 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.585402    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca\") pod \"66847664-78a5-40cf-90fd-aeaa9cd993d7\" (UID: \"66847664-78a5-40cf-90fd-aeaa9cd993d7\") "
Jul 07 16:54:02.585715 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.585690    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66847664-78a5-40cf-90fd-aeaa9cd993d7" (UID: "66847664-78a5-40cf-90fd-aeaa9cd993d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:54:02.587647 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.587624    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn" (OuterVolumeSpecName: "kube-api-access-lxdgn") pod "66847664-78a5-40cf-90fd-aeaa9cd993d7" (UID: "66847664-78a5-40cf-90fd-aeaa9cd993d7"). InnerVolumeSpecName "kube-api-access-lxdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:54:02.587739 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.587688    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66847664-78a5-40cf-90fd-aeaa9cd993d7" (UID: "66847664-78a5-40cf-90fd-aeaa9cd993d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:54:02.686535 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.686510    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdgn\" (UniqueName: \"kubernetes.io/projected/66847664-78a5-40cf-90fd-aeaa9cd993d7-kube-api-access-lxdgn\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:54:02.686535 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.686534    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66847664-78a5-40cf-90fd-aeaa9cd993d7-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:54:02.686715 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:02.686543    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66847664-78a5-40cf-90fd-aeaa9cd993d7-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:54:03.045119 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.045047    2565 generic.go:356] "Generic (PLEG): container finished" podID="66847664-78a5-40cf-90fd-aeaa9cd993d7" containerID="5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e" exitCode=0
Jul 07 16:54:03.045119 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.045101    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"
Jul 07 16:54:03.045276 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.045128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" event={"ID":"66847664-78a5-40cf-90fd-aeaa9cd993d7","Type":"ContainerDied","Data":"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"}
Jul 07 16:54:03.045276 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.045185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4" event={"ID":"66847664-78a5-40cf-90fd-aeaa9cd993d7","Type":"ContainerDied","Data":"daec79334256bb646ee436c900610cc757625f9f52264ebceed04d78987fa8a4"}
Jul 07 16:54:03.045276 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.045209    2565 scope.go:117] "RemoveContainer" containerID="5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"
Jul 07 16:54:03.054172 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.054155    2565 scope.go:117] "RemoveContainer" containerID="5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"
Jul 07 16:54:03.054429 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:54:03.054410    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e\": container with ID starting with 5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e not found: ID does not exist" containerID="5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"
Jul 07 16:54:03.054498 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.054436    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e"} err="failed to get container status \"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e\": rpc error: code = NotFound desc = could not find container \"5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e\": container with ID starting with 5fd9057a1182b838caf9f753e63ad11f0f46dd5db4d8ee9fd3948afaace1193e not found: ID does not exist"
Jul 07 16:54:03.069680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.069654    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:54:03.078333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:03.078311    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-f2f03512-7c9df5686c-8f7q4"]
Jul 07 16:54:04.094349 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:54:04.094320    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66847664-78a5-40cf-90fd-aeaa9cd993d7" path="/var/lib/kubelet/pods/66847664-78a5-40cf-90fd-aeaa9cd993d7/volumes"
Jul 07 16:55:47.819146 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.819111    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:55:47.819535 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.819475    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66847664-78a5-40cf-90fd-aeaa9cd993d7" containerName="maas-api"
Jul 07 16:55:47.819535 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.819485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66847664-78a5-40cf-90fd-aeaa9cd993d7" containerName="maas-api"
Jul 07 16:55:47.819610 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.819551    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66847664-78a5-40cf-90fd-aeaa9cd993d7" containerName="maas-api"
Jul 07 16:55:47.822462 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.822445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.825575 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.825553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79f2ee03-openshift-default-dockercfg-7lw9f\""
Jul 07 16:55:47.844438 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.844415    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:55:47.861211 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861334 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861334 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw2z\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861446 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861446 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861552 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861552 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861552 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.861701 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.861593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963013 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.962978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw2z\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963171 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963427 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963427 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963427 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963427 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963407    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963609 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963458    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963609 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963480    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.963693 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.963645    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.964115 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.964098    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.965332 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.965304    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.965586 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.965569    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.973140 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.973113    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:47.973483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:47.973461    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw2z\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z\") pod \"e2e-mt-79f2ee03-openshift-default-795954655-lvkn7\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") " pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:48.132053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.131992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:48.268960 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.268939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:55:48.271098 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:55:48.271072    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bce2e23_af9b_4c9c_a0bf_da99f151a38c.slice/crio-ae08d133e2f54e33d563b2f734888a0d1b3c6b45d1740e8e4bfba9a08c6c5b9d WatchSource:0}: Error finding container ae08d133e2f54e33d563b2f734888a0d1b3c6b45d1740e8e4bfba9a08c6c5b9d: Status 404 returned error can't find the container with id ae08d133e2f54e33d563b2f734888a0d1b3c6b45d1740e8e4bfba9a08c6c5b9d
Jul 07 16:55:48.273034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.273003    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:55:48.273103 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.273063    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:55:48.273103 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.273092    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:55:48.581420 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.581387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerStarted","Data":"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"}
Jul 07 16:55:48.581576 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.581446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerStarted","Data":"ae08d133e2f54e33d563b2f734888a0d1b3c6b45d1740e8e4bfba9a08c6c5b9d"}
Jul 07 16:55:48.618964 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:48.618915    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podStartSLOduration=1.618899654 podStartE2EDuration="1.618899654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:55:48.617068679 +0000 UTC m=+1947.149985342" watchObservedRunningTime="2026-07-07 16:55:48.618899654 +0000 UTC m=+1947.151816296"
Jul 07 16:55:49.133129 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:49.133097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:49.138004 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:49.137980    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:49.586741 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:49.586718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:55:49.587612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:55:49.587594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:04.588537 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.588505    2565 patch_prober.go:28] interesting pod/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 07 16:56:04.588940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.588561    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 07 16:56:04.663129 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.663100    2565 generic.go:356] "Generic (PLEG): container finished" podID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerID="504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09" exitCode=0
Jul 07 16:56:04.663273 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.663176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerDied","Data":"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"}
Jul 07 16:56:04.663519 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.663507    2565 scope.go:117] "RemoveContainer" containerID="504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"
Jul 07 16:56:04.664542 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.664514    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:04.664601 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.664590    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:04.664652 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:04.664637    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:05.669178 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:05.669143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerStarted","Data":"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"}
Jul 07 16:56:06.132593 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:06.132556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:06.132962 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:06.132928    2565 patch_prober.go:28] interesting pod/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 07 16:56:06.133076 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:06.132992    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 07 16:56:07.132707 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:07.132672    2565 patch_prober.go:28] interesting pod/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 07 16:56:07.133253 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:07.133223    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 07 16:56:08.132605 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:08.132572    2565 patch_prober.go:28] interesting pod/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 07 16:56:08.132812 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:08.132623    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 07 16:56:09.133162 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:09.133128    2565 patch_prober.go:28] interesting pod/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 07 16:56:09.133612 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:09.133208    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 07 16:56:10.136335 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:10.136308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:10.136700 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:10.136574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:10.137302 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:10.137277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:11.760895 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:11.760866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:56:12.703013 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:12.702977    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy" containerID="cri-o://11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a" gracePeriod=30
Jul 07 16:56:17.950669 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:17.950646    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:18.021150 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021269 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vw2z\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021269 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021269 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021429 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021397    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs\") pod \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\" (UID: \"9bce2e23-af9b-4c9c-a0bf-da99f151a38c\") "
Jul 07 16:56:18.021669 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021634    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:18.021830 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021754    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:56:18.021927 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021903    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:18.021979 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.021917    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:18.022191 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.022160    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:18.023288 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.023264    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:56:18.023492 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.023477    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z" (OuterVolumeSpecName: "kube-api-access-8vw2z") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "kube-api-access-8vw2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:56:18.023561 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.023493    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:18.023561 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.023540    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9bce2e23-af9b-4c9c-a0bf-da99f151a38c" (UID: "9bce2e23-af9b-4c9c-a0bf-da99f151a38c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:56:18.122600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122572    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122593    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vw2z\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-kube-api-access-8vw2z\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122600 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122603    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122611    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122618    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122625    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122632    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122639    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.122738 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.122646    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bce2e23-af9b-4c9c-a0bf-da99f151a38c-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:56:18.735133 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.735105    2565 generic.go:356] "Generic (PLEG): container finished" podID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerID="11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a" exitCode=0
Jul 07 16:56:18.735254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.735169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerDied","Data":"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"}
Jul 07 16:56:18.735254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.735181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"
Jul 07 16:56:18.735254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.735215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7" event={"ID":"9bce2e23-af9b-4c9c-a0bf-da99f151a38c","Type":"ContainerDied","Data":"ae08d133e2f54e33d563b2f734888a0d1b3c6b45d1740e8e4bfba9a08c6c5b9d"}
Jul 07 16:56:18.735254 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.735239    2565 scope.go:117] "RemoveContainer" containerID="11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"
Jul 07 16:56:18.745042 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.745023    2565 scope.go:117] "RemoveContainer" containerID="504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"
Jul 07 16:56:18.753654 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.753635    2565 scope.go:117] "RemoveContainer" containerID="11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"
Jul 07 16:56:18.753985 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:56:18.753963    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a\": container with ID starting with 11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a not found: ID does not exist" containerID="11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"
Jul 07 16:56:18.754042 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.753995    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a"} err="failed to get container status \"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a\": rpc error: code = NotFound desc = could not find container \"11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a\": container with ID starting with 11d53c4a244c238c2d62b5bb6253153f9c2486ddb0f96712933f37b4d240034a not found: ID does not exist"
Jul 07 16:56:18.754042 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.754016    2565 scope.go:117] "RemoveContainer" containerID="504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"
Jul 07 16:56:18.754276 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:56:18.754257    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09\": container with ID starting with 504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09 not found: ID does not exist" containerID="504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"
Jul 07 16:56:18.754333 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.754286    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09"} err="failed to get container status \"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09\": rpc error: code = NotFound desc = could not find container \"504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09\": container with ID starting with 504d0ac59282cbef2bece37efaea69d340033a8c6d848698cb210b66641fce09 not found: ID does not exist"
Jul 07 16:56:18.759629 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.759602    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:56:18.774758 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:18.774732    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f2ee03-openshift-default-795954655-lvkn7"]
Jul 07 16:56:20.102463 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:20.102431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" path="/var/lib/kubelet/pods/9bce2e23-af9b-4c9c-a0bf-da99f151a38c/volumes"
Jul 07 16:56:33.323427 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323398    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:56:33.323907 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:33.323907 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323753    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:33.323907 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323766    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:33.323907 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323817    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:33.324053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.323913    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:33.325730 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.325712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.329966 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.329949    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-51ea3c78-openshift-default-dockercfg-5htcd\""
Jul 07 16:56:33.363766 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.363742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:56:33.436862 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.436862 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437053 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.436983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2k5\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.437074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.437126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.437176 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.437156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538225 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2k5\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538366 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538568 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538758 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538629    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.538758 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.538688    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.539100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.539080    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.539100 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.539091    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.539269 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.539254    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.540451 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.540433    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.541080 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.541065    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.560162 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.560140    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.560962 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.560946    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2k5\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5\") pod \"e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") " pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.636520 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.636467    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:33.787604 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.787578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:56:33.788745 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:56:33.788718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a98f21_33de_4f43_8a5f_fdcd42f673d9.slice/crio-f77a8ca7a199fa215f76b122c958816ea51f99ffa1add833ae927c43a76cdba0 WatchSource:0}: Error finding container f77a8ca7a199fa215f76b122c958816ea51f99ffa1add833ae927c43a76cdba0: Status 404 returned error can't find the container with id f77a8ca7a199fa215f76b122c958816ea51f99ffa1add833ae927c43a76cdba0
Jul 07 16:56:33.791821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.791183    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:33.791821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.791362    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:33.791821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.791447    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:33.812051 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:33.812024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerStarted","Data":"f77a8ca7a199fa215f76b122c958816ea51f99ffa1add833ae927c43a76cdba0"}
Jul 07 16:56:34.818379 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:34.818342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerStarted","Data":"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"}
Jul 07 16:56:34.847935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:34.847896    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podStartSLOduration=1.8478810289999998 podStartE2EDuration="1.847881029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:34.845272844 +0000 UTC m=+1993.378189502" watchObservedRunningTime="2026-07-07 16:56:34.847881029 +0000 UTC m=+1993.380797671"
Jul 07 16:56:35.637274 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:35.637243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:35.642395 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:35.642370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:35.823189 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:35.823166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:35.824005 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:35.823988    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:36.987001 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:36.986973    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:56:36.987538 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:36.987519    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bce2e23-af9b-4c9c-a0bf-da99f151a38c" containerName="istio-proxy"
Jul 07 16:56:36.989616 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:36.989597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:36.993851 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:36.993828    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-51ea3c78-serving-cert\""
Jul 07 16:56:36.993943 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:36.993902    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\""
Jul 07 16:56:37.013566 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.013542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:56:37.166072 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.166037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.166250 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.166088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.166250 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.166119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfzr\" (UniqueName: \"kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.267118 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.267047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.267118 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.267080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.267118 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.267104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfzr\" (UniqueName: \"kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.267728 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.267707    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.269447 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.269416    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.288670 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.288647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfzr\" (UniqueName: \"kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr\") pod \"maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") " pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.300180 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.300162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:37.493328 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.493301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:56:37.494153 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:56:37.494130    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737708f3_76f5_4b1d_8db8_4ef893fae6d0.slice/crio-f5d28a172492b34edd71e9e457baee989ce7cabdafa37478e02cbeca38efbeb7 WatchSource:0}: Error finding container f5d28a172492b34edd71e9e457baee989ce7cabdafa37478e02cbeca38efbeb7: Status 404 returned error can't find the container with id f5d28a172492b34edd71e9e457baee989ce7cabdafa37478e02cbeca38efbeb7
Jul 07 16:56:37.833550 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:37.833525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" event={"ID":"737708f3-76f5-4b1d-8db8-4ef893fae6d0","Type":"ContainerStarted","Data":"f5d28a172492b34edd71e9e457baee989ce7cabdafa37478e02cbeca38efbeb7"}
Jul 07 16:56:38.839825 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:38.839769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" event={"ID":"737708f3-76f5-4b1d-8db8-4ef893fae6d0","Type":"ContainerStarted","Data":"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"}
Jul 07 16:56:38.840206 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:38.839894    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:38.884284 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:38.884242    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" podStartSLOduration=2.56851389 podStartE2EDuration="2.884230625s" totalImagesPullingTime="315.716735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:38.88299938 +0000 UTC m=+1997.415916030" watchObservedRunningTime="2026-07-07 16:56:38.884230625 +0000 UTC m=+1997.417147267"
Jul 07 16:56:44.851051 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:44.851021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:56:47.675544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.675516    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:56:47.677992 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.677977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.680852 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.680828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-51ea3c78\"/\"openshift-service-ca.crt\""
Jul 07 16:56:47.682187 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.682168    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-51ea3c78\"/\"default-dockercfg-pcjcz\""
Jul 07 16:56:47.682300 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.682209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-51ea3c78\"/\"kube-root-ca.crt\""
Jul 07 16:56:47.682300 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.682173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-51ea3c78\"/\"e2e-lifecycle-model-51ea3c78-kserve-self-signed-certs\""
Jul 07 16:56:47.690000 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.689982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:56:47.761094 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.761238 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56q9\" (UniqueName: \"kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.761238 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.761238 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.761344 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.761385 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.761370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862149 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56q9\" (UniqueName: \"kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862325 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862325 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862325 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862325 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862531 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862581    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862669 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862649    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.862669 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.862661    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.864727 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.864698    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.864874 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.864837    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.870991 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.870962    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56q9\" (UniqueName: \"kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9\") pod \"e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") " pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:47.988357 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:47.988269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:48.130974 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:48.130944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:56:48.132511 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:56:48.132483    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97e889dc_400b_4710_a720_a3fee24b0d93.slice/crio-2ac54065cfa75bd37ce5510da7ce0b2caff3521c5994d45578fd994e02b7a697 WatchSource:0}: Error finding container 2ac54065cfa75bd37ce5510da7ce0b2caff3521c5994d45578fd994e02b7a697: Status 404 returned error can't find the container with id 2ac54065cfa75bd37ce5510da7ce0b2caff3521c5994d45578fd994e02b7a697
Jul 07 16:56:48.889588 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:48.889554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" event={"ID":"97e889dc-400b-4710-a720-a3fee24b0d93","Type":"ContainerStarted","Data":"67811a5bb547e7c1286cd0e56c1176ffbbbdb1d4464129c93d754c0f0878862f"}
Jul 07 16:56:48.889966 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:48.889593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" event={"ID":"97e889dc-400b-4710-a720-a3fee24b0d93","Type":"ContainerStarted","Data":"2ac54065cfa75bd37ce5510da7ce0b2caff3521c5994d45578fd994e02b7a697"}
Jul 07 16:56:50.824529 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.824496    2565 patch_prober.go:28] interesting pod/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 07 16:56:50.824997 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.824573    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 07 16:56:50.900608 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.900584    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerID="4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1" exitCode=0
Jul 07 16:56:50.900741 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.900665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerDied","Data":"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"}
Jul 07 16:56:50.901010 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.900996    2565 scope.go:117] "RemoveContainer" containerID="4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"
Jul 07 16:56:50.901989 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.901963    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:50.902063 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.902026    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:50.902063 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.902059    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:56:50.942684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:50.942632    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" podStartSLOduration=3.942614498 podStartE2EDuration="3.942614498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:48.914943143 +0000 UTC m=+2007.447859785" watchObservedRunningTime="2026-07-07 16:56:50.942614498 +0000 UTC m=+2009.475531142"
Jul 07 16:56:51.908064 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:51.908020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerStarted","Data":"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"}
Jul 07 16:56:52.637516 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:52.637482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:52.637752 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:52.637731    2565 patch_prober.go:28] interesting pod/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 07 16:56:52.637867 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:52.637819    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 07 16:56:53.637438 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:53.637404    2565 patch_prober.go:28] interesting pod/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 07 16:56:53.637809 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:53.637456    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 07 16:56:54.637533 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:54.637501    2565 patch_prober.go:28] interesting pod/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 07 16:56:54.637926 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:54.637552    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 07 16:56:55.641019 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:55.640984    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:55.641471 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:55.641362    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:55.642077 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:55.642056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:56:57.989255 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:57.989219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:57.989718 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:57.989270    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:58.001206 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:58.001175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:56:58.966971 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:56:58.966945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:57:09.230370 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.230322    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:57:09.230971 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.230548    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" podUID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" containerName="maas-api" containerID="cri-o://d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a" gracePeriod=30
Jul 07 16:57:09.464896 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.464872    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:57:09.532870 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.532811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca\") pod \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") "
Jul 07 16:57:09.532870 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.532850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls\") pod \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") "
Jul 07 16:57:09.533005 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.532886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwfzr\" (UniqueName: \"kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr\") pod \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\" (UID: \"737708f3-76f5-4b1d-8db8-4ef893fae6d0\") "
Jul 07 16:57:09.533219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.533194    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "737708f3-76f5-4b1d-8db8-4ef893fae6d0" (UID: "737708f3-76f5-4b1d-8db8-4ef893fae6d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:57:09.534821 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.534775    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr" (OuterVolumeSpecName: "kube-api-access-jwfzr") pod "737708f3-76f5-4b1d-8db8-4ef893fae6d0" (UID: "737708f3-76f5-4b1d-8db8-4ef893fae6d0"). InnerVolumeSpecName "kube-api-access-jwfzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:57:09.534968 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.534836    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "737708f3-76f5-4b1d-8db8-4ef893fae6d0" (UID: "737708f3-76f5-4b1d-8db8-4ef893fae6d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:57:09.634395 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.634364    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737708f3-76f5-4b1d-8db8-4ef893fae6d0-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:09.634395 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.634390    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737708f3-76f5-4b1d-8db8-4ef893fae6d0-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:09.634532 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:09.634404    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfzr\" (UniqueName: \"kubernetes.io/projected/737708f3-76f5-4b1d-8db8-4ef893fae6d0-kube-api-access-jwfzr\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:10.014972 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.014945    2565 generic.go:356] "Generic (PLEG): container finished" podID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" containerID="d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a" exitCode=0
Jul 07 16:57:10.015130 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.014996    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"
Jul 07 16:57:10.015130 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.015028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" event={"ID":"737708f3-76f5-4b1d-8db8-4ef893fae6d0","Type":"ContainerDied","Data":"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"}
Jul 07 16:57:10.015249 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.015158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj" event={"ID":"737708f3-76f5-4b1d-8db8-4ef893fae6d0","Type":"ContainerDied","Data":"f5d28a172492b34edd71e9e457baee989ce7cabdafa37478e02cbeca38efbeb7"}
Jul 07 16:57:10.015294 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.015215    2565 scope.go:117] "RemoveContainer" containerID="d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"
Jul 07 16:57:10.024327 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.024309    2565 scope.go:117] "RemoveContainer" containerID="d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"
Jul 07 16:57:10.024681 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:57:10.024661    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a\": container with ID starting with d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a not found: ID does not exist" containerID="d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"
Jul 07 16:57:10.024750 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.024688    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a"} err="failed to get container status \"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a\": rpc error: code = NotFound desc = could not find container \"d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a\": container with ID starting with d2ddac2b76c659dda47c1b8225e9a1275ea5d2c6a9d03fcf88b4c339daabe68a not found: ID does not exist"
Jul 07 16:57:10.039690 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.039673    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:57:10.044776 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.044758    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-51ea3c78-679d9c57df-m6fwj"]
Jul 07 16:57:10.096286 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:10.096263    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" path="/var/lib/kubelet/pods/737708f3-76f5-4b1d-8db8-4ef893fae6d0/volumes"
Jul 07 16:57:19.943057 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:19.943023    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:57:34.962605 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:34.962543    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" podUID="97e889dc-400b-4710-a720-a3fee24b0d93" containerName="main" containerID="cri-o://67811a5bb547e7c1286cd0e56c1176ffbbbdb1d4464129c93d754c0f0878862f" gracePeriod=45
Jul 07 16:57:35.146860 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.146823    2565 generic.go:356] "Generic (PLEG): container finished" podID="97e889dc-400b-4710-a720-a3fee24b0d93" containerID="67811a5bb547e7c1286cd0e56c1176ffbbbdb1d4464129c93d754c0f0878862f" exitCode=0
Jul 07 16:57:35.147014 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.146917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" event={"ID":"97e889dc-400b-4710-a720-a3fee24b0d93","Type":"ContainerDied","Data":"67811a5bb547e7c1286cd0e56c1176ffbbbdb1d4464129c93d754c0f0878862f"}
Jul 07 16:57:35.207441 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.207421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:57:35.333081 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333115    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333178    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f56q9\" (UniqueName: \"kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333227 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs\") pod \"97e889dc-400b-4710-a720-a3fee24b0d93\" (UID: \"97e889dc-400b-4710-a720-a3fee24b0d93\") "
Jul 07 16:57:35.333445 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333338    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache" (OuterVolumeSpecName: "model-cache") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:57:35.333445 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333352    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home" (OuterVolumeSpecName: "home") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:57:35.333445 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333360    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:57:35.333571 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333546    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:35.333571 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333564    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:35.333684 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.333589    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:35.335283 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.335262    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm" (OuterVolumeSpecName: "dshm") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:57:35.335358 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.335275    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:57:35.335358 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.335290    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9" (OuterVolumeSpecName: "kube-api-access-f56q9") pod "97e889dc-400b-4710-a720-a3fee24b0d93" (UID: "97e889dc-400b-4710-a720-a3fee24b0d93"). InnerVolumeSpecName "kube-api-access-f56q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:57:35.434873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.434845    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e889dc-400b-4710-a720-a3fee24b0d93-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:35.434873 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.434869    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e889dc-400b-4710-a720-a3fee24b0d93-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:35.435008 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:35.434882    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f56q9\" (UniqueName: \"kubernetes.io/projected/97e889dc-400b-4710-a720-a3fee24b0d93-kube-api-access-f56q9\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:57:36.153034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:36.153010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8" event={"ID":"97e889dc-400b-4710-a720-a3fee24b0d93","Type":"ContainerDied","Data":"2ac54065cfa75bd37ce5510da7ce0b2caff3521c5994d45578fd994e02b7a697"}
Jul 07 16:57:36.153034 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:36.153027    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"
Jul 07 16:57:36.153399 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:36.153048    2565 scope.go:117] "RemoveContainer" containerID="67811a5bb547e7c1286cd0e56c1176ffbbbdb1d4464129c93d754c0f0878862f"
Jul 07 16:57:36.176205 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:36.176182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:57:36.178770 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:36.178739    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-51ea3c78/e2e-lifecycle-model-51ea3c78-kserve-79c6c47655-ntcm8"]
Jul 07 16:57:38.097480 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:57:38.097432    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e889dc-400b-4710-a720-a3fee24b0d93" path="/var/lib/kubelet/pods/97e889dc-400b-4710-a720-a3fee24b0d93/volumes"
Jul 07 16:58:04.989530 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:04.989444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:58:04.990064 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:04.989771    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy" containerID="cri-o://6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd" gracePeriod=30
Jul 07 16:58:10.227929 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.227905    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:58:10.312219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww2k5\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312219 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312315    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312375    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312456 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data\") pod \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\" (UID: \"f8a98f21-33de-4f43-8a5f-fdcd42f673d9\") "
Jul 07 16:58:10.312769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312472    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:10.312769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312485    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:10.312769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312721    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:10.312769 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312728    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:10.312982 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312821    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.312982 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.312847    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.313622 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.313589    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:10.314525 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.314493    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:10.314644 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.314625    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:10.314858 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.314832    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 16:58:10.314973 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.314950    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5" (OuterVolumeSpecName: "kube-api-access-ww2k5") pod "f8a98f21-33de-4f43-8a5f-fdcd42f673d9" (UID: "f8a98f21-33de-4f43-8a5f-fdcd42f673d9"). InnerVolumeSpecName "kube-api-access-ww2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:10.316480 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.316453    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerID="6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd" exitCode=0
Jul 07 16:58:10.316572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.316480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerDied","Data":"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"}
Jul 07 16:58:10.316572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.316494    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"
Jul 07 16:58:10.316572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.316534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz" event={"ID":"f8a98f21-33de-4f43-8a5f-fdcd42f673d9","Type":"ContainerDied","Data":"f77a8ca7a199fa215f76b122c958816ea51f99ffa1add833ae927c43a76cdba0"}
Jul 07 16:58:10.316572 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.316555    2565 scope.go:117] "RemoveContainer" containerID="6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"
Jul 07 16:58:10.331201 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.331185    2565 scope.go:117] "RemoveContainer" containerID="4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"
Jul 07 16:58:10.338451 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.338437    2565 scope.go:117] "RemoveContainer" containerID="6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"
Jul 07 16:58:10.338741 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:58:10.338711    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd\": container with ID starting with 6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd not found: ID does not exist" containerID="6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"
Jul 07 16:58:10.338819 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.338749    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd"} err="failed to get container status \"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd\": rpc error: code = NotFound desc = could not find container \"6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd\": container with ID starting with 6376b0b11850ff9386580b1eaa6bc254eccd6929d9068214a7229bf9c2099bcd not found: ID does not exist"
Jul 07 16:58:10.338819 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.338765    2565 scope.go:117] "RemoveContainer" containerID="4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"
Jul 07 16:58:10.339032 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:58:10.339017    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1\": container with ID starting with 4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1 not found: ID does not exist" containerID="4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"
Jul 07 16:58:10.339081 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.339038    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1"} err="failed to get container status \"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1\": rpc error: code = NotFound desc = could not find container \"4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1\": container with ID starting with 4ace536f041486f215478adb204e084d8cfa4ce56b74553864e3865d508539b1 not found: ID does not exist"
Jul 07 16:58:10.374828 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.374809    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:58:10.404483 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.404457    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-51ea3c78-openshift-default-7db69f8d7-r5ggz"]
Jul 07 16:58:10.413658 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413635    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413658 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413656    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413664    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413671    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413680    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2k5\" (UniqueName: \"kubernetes.io/projected/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-kube-api-access-ww2k5\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413688    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.413773 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:10.413696    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a98f21-33de-4f43-8a5f-fdcd42f673d9-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 16:58:12.096168 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:58:12.096135    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" path="/var/lib/kubelet/pods/f8a98f21-33de-4f43-8a5f-fdcd42f673d9/volumes"
Jul 07 16:59:03.492071 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492035    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492373    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492384    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492393    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492398    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492411    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e889dc-400b-4710-a720-a3fee24b0d93" containerName="main"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492416    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e889dc-400b-4710-a720-a3fee24b0d93" containerName="main"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" containerName="maas-api"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492429    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" containerName="maas-api"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492489    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="737708f3-76f5-4b1d-8db8-4ef893fae6d0" containerName="maas-api"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492495    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e889dc-400b-4710-a720-a3fee24b0d93" containerName="main"
Jul 07 16:59:03.492499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492504    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.492881 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.492510    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a98f21-33de-4f43-8a5f-fdcd42f673d9" containerName="istio-proxy"
Jul 07 16:59:03.495422 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.495398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.498709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.498691    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1b64ca-openshift-default-dockercfg-9k4zj\""
Jul 07 16:59:03.517308 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.517284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 16:59:03.606437 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84p2\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606609 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606609 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606609 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606736 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606736 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606736 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.606935 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.606853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708115 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708263 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708263 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708383 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708438 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708493 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84p2\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708537    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708597 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708580    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.708804 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.708667    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.709188 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.709152    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.709400 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.709378    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.710923 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.710903    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.711163 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.711143    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.719302 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.719277    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84p2\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.719541 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.719521    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token\") pod \"e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") " pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:03.805346 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:03.805288    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:04.150390 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.150362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 16:59:04.151872 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:59:04.151848    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2d106b_3af4_429c_b78e_6a6794085fdb.slice/crio-076095c39a5caae62d0af21ea583419f6b30e0712782843466410f424494c077 WatchSource:0}: Error finding container 076095c39a5caae62d0af21ea583419f6b30e0712782843466410f424494c077: Status 404 returned error can't find the container with id 076095c39a5caae62d0af21ea583419f6b30e0712782843466410f424494c077
Jul 07 16:59:04.153852 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.153828    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:04.153906 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.153883    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:04.153940 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.153912    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:04.576517 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.576479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerStarted","Data":"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"}
Jul 07 16:59:04.576962 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.576530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerStarted","Data":"076095c39a5caae62d0af21ea583419f6b30e0712782843466410f424494c077"}
Jul 07 16:59:04.607680 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.607629    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podStartSLOduration=1.607609479 podStartE2EDuration="1.607609479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:04.606191461 +0000 UTC m=+2143.139108103" watchObservedRunningTime="2026-07-07 16:59:04.607609479 +0000 UTC m=+2143.140526123"
Jul 07 16:59:04.805947 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:04.805917    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:05.810092 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:05.810060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:06.586860 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.586825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:06.587921 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.587897    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:06.756735 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.756700    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 16:59:06.760470 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.760450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:06.774341 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.774322    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\""
Jul 07 16:59:06.776056 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.776038    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-1b64ca-serving-cert\""
Jul 07 16:59:06.782893 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.782871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 16:59:06.935426 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.935343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:06.935426 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.935381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v86\" (UniqueName: \"kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:06.935986 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:06.935455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.036878 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.036849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.036878 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.036882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v86\" (UniqueName: \"kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.037062 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.036943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.037452 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.037433    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.039178 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.039161    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.046476 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.046457    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v86\" (UniqueName: \"kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86\") pod \"maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") " pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.070020 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.070002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:07.211374 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.211349    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 16:59:07.213840 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:59:07.213811    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea67efc_4e3c_491f_b9e5_719e172af3fc.slice/crio-04c6f16bca96be13c4d86570fab5f56c8732e84ead92c195f4e10fb951cd0897 WatchSource:0}: Error finding container 04c6f16bca96be13c4d86570fab5f56c8732e84ead92c195f4e10fb951cd0897: Status 404 returned error can't find the container with id 04c6f16bca96be13c4d86570fab5f56c8732e84ead92c195f4e10fb951cd0897
Jul 07 16:59:07.215661 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.215642    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:59:07.592857 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:07.592826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" event={"ID":"6ea67efc-4e3c-491f-b9e5-719e172af3fc","Type":"ContainerStarted","Data":"04c6f16bca96be13c4d86570fab5f56c8732e84ead92c195f4e10fb951cd0897"}
Jul 07 16:59:08.598933 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:08.598888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" event={"ID":"6ea67efc-4e3c-491f-b9e5-719e172af3fc","Type":"ContainerStarted","Data":"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"}
Jul 07 16:59:08.599305 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:08.598963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:08.627230 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:08.627188    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" podStartSLOduration=1.948913643 podStartE2EDuration="2.627149834s" totalImagesPullingTime="678.236191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:08.625537432 +0000 UTC m=+2147.158454077" watchObservedRunningTime="2026-07-07 16:59:08.627149834 +0000 UTC m=+2147.160066476"
Jul 07 16:59:14.628980 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:14.628097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 16:59:15.756803 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.756751    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 16:59:15.760996 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.760971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.764331 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.764309    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-4b5d6a-serving-cert\""
Jul 07 16:59:15.779173 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.779149    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 16:59:15.814204 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.814175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4w6\" (UniqueName: \"kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.814474 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.814454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.814628 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.814610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.915709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.915670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.915913 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.915729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.915913 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.915765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4w6\" (UniqueName: \"kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.916231 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:59:15.916209    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-4b5d6a-serving-cert: secret "maas-api-e2e-inf-b-4b5d6a-serving-cert" not found
Jul 07 16:59:15.916318 ip-10-0-140-28 kubenswrapper[2565]: E0707 16:59:15.916277    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls podName:3b902c4c-7720-4a72-ac58-3beb2569ec7a nodeName:}" failed. No retries permitted until 2026-07-07 16:59:16.416257489 +0000 UTC m=+2154.949174128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls") pod "maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" (UID: "3b902c4c-7720-4a72-ac58-3beb2569ec7a") : secret "maas-api-e2e-inf-b-4b5d6a-serving-cert" not found
Jul 07 16:59:15.917188 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.917158    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:15.931421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:15.931398    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4w6\" (UniqueName: \"kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:16.420371 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:16.420331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:16.423504 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:16.423482    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") " pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:16.671536 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:16.671454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:16.859289 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:16.859266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 16:59:16.861370 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:59:16.861334    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b902c4c_7720_4a72_ac58_3beb2569ec7a.slice/crio-6fde5b9eb0e8043ff9e84b2c50904033297030f3475cb51fc29ce75c9b996726 WatchSource:0}: Error finding container 6fde5b9eb0e8043ff9e84b2c50904033297030f3475cb51fc29ce75c9b996726: Status 404 returned error can't find the container with id 6fde5b9eb0e8043ff9e84b2c50904033297030f3475cb51fc29ce75c9b996726
Jul 07 16:59:17.651380 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:17.651341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" event={"ID":"3b902c4c-7720-4a72-ac58-3beb2569ec7a","Type":"ContainerStarted","Data":"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"}
Jul 07 16:59:17.651558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:17.651418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" event={"ID":"3b902c4c-7720-4a72-ac58-3beb2569ec7a","Type":"ContainerStarted","Data":"6fde5b9eb0e8043ff9e84b2c50904033297030f3475cb51fc29ce75c9b996726"}
Jul 07 16:59:17.651558 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:17.651439    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:19.662737 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.662713    2565 generic.go:356] "Generic (PLEG): container finished" podID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerID="bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443" exitCode=0
Jul 07 16:59:19.663054 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.662772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerDied","Data":"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"}
Jul 07 16:59:19.663194 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.663168    2565 scope.go:117] "RemoveContainer" containerID="bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"
Jul 07 16:59:19.664107 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.664074    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:19.664196 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.664149    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:19.664196 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.664188    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 16:59:19.712709 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:19.712666    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" podStartSLOduration=4.395144357 podStartE2EDuration="4.712651616s" totalImagesPullingTime="317.507259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:17.691700025 +0000 UTC m=+2156.224616668" watchObservedRunningTime="2026-07-07 16:59:19.712651616 +0000 UTC m=+2158.245568280"
Jul 07 16:59:20.669860 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:20.669825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerStarted","Data":"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"}
Jul 07 16:59:20.805822 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:20.805771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:20.806944 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:20.806909    2565 patch_prober.go:28] interesting pod/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 07 16:59:20.807090 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:20.806966    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 07 16:59:21.587712 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:21.587677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:21.805824 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:21.805767    2565 patch_prober.go:28] interesting pod/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 07 16:59:21.806262 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:21.805846    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 07 16:59:22.805857 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:22.805823    2565 patch_prober.go:28] interesting pod/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 07 16:59:22.806393 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:22.806361    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 07 16:59:23.661281 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:23.661257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 16:59:23.810499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:23.810105    2565 patch_prober.go:28] interesting pod/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 07 16:59:23.810499 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:23.810163    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 07 16:59:24.101336 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.101310    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 16:59:24.104036 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.104018    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.108101 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.108077    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b64ca\"/\"default-dockercfg-dh8g8\""
Jul 07 16:59:24.108220 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.108165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b64ca\"/\"kube-root-ca.crt\""
Jul 07 16:59:24.108438 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.108419    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b64ca\"/\"openshift-service-ca.crt\""
Jul 07 16:59:24.108653 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.108635    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b64ca\"/\"test-model-1b64ca-kserve-self-signed-certs\""
Jul 07 16:59:24.128119 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.128094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 16:59:24.204306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.204306 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.204544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.204544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.204544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvhf\" (UniqueName: \"kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.204544 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.204496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305564 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305564 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvhf\" (UniqueName: \"kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.305772 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.305738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.306337 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.306310    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.306504 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.306298    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.306591 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.306321    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.308697 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.308651    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.308863 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.308842    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.317365 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.317333    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvhf\" (UniqueName: \"kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf\") pod \"test-model-1b64ca-kserve-57b4d688b7-xs6kr\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") " pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.416607 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.416545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:24.553008 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.552971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 16:59:24.553954 ip-10-0-140-28 kubenswrapper[2565]: W0707 16:59:24.553931    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fb305a0_6dc3_4599_9ecf_db9538e6832e.slice/crio-08a5c6c517380d131851dccb70011891f1f1c4abddb58ff621007acd5677144a WatchSource:0}: Error finding container 08a5c6c517380d131851dccb70011891f1f1c4abddb58ff621007acd5677144a: Status 404 returned error can't find the container with id 08a5c6c517380d131851dccb70011891f1f1c4abddb58ff621007acd5677144a
Jul 07 16:59:24.689432 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.689367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" event={"ID":"9fb305a0-6dc3-4599-9ecf-db9538e6832e","Type":"ContainerStarted","Data":"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"}
Jul 07 16:59:24.689432 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.689406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" event={"ID":"9fb305a0-6dc3-4599-9ecf-db9538e6832e","Type":"ContainerStarted","Data":"08a5c6c517380d131851dccb70011891f1f1c4abddb58ff621007acd5677144a"}
Jul 07 16:59:24.711603 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.711561    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" podStartSLOduration=0.711547371 podStartE2EDuration="711.547371ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:24.709189922 +0000 UTC m=+2163.242106565" watchObservedRunningTime="2026-07-07 16:59:24.711547371 +0000 UTC m=+2163.244464013"
Jul 07 16:59:24.810848 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.810825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:24.811756 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:24.811737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 16:59:34.416927 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:34.416898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:34.417421 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:34.416938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:34.428331 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:34.428311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 16:59:34.755215 ip-10-0-140-28 kubenswrapper[2565]: I0707 16:59:34.755149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 17:00:16.691452 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.691420    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:00:16.695006 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.694982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.699698 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.699573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-37478f\"/\"default-dockercfg-jst9m\""
Jul 07 17:00:16.699698 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.699644    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-37478f\"/\"test-model-37478f-kserve-self-signed-certs\""
Jul 07 17:00:16.699698 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.699671    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-37478f\"/\"openshift-service-ca.crt\""
Jul 07 17:00:16.699967 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.699648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-37478f\"/\"kube-root-ca.crt\""
Jul 07 17:00:16.711276 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.711250    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:00:16.723115 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmqf\" (UniqueName: \"kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.723273 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.723273 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.723273 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.723273 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.723273 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.723221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824261 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824438 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824438 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824438 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824613 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmqf\" (UniqueName: \"kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824613 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824718 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824659    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824834 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824817    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.824973 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.824952    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.826496 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.826477    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.826747 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.826731    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:16.842967 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:16.842947    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmqf\" (UniqueName: \"kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf\") pod \"test-model-37478f-kserve-dc8df9df9-fsjxb\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") " pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:17.004616 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:17.004556    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:17.203589 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:17.203560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:00:17.205677 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:00:17.205645    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2e5a3f_f19e_4d6b_99d5_5c86e20b53fa.slice/crio-ece6e6b8b09727be58549ee097761db91e89799b2434d3ca3b9f2e0baf72f545 WatchSource:0}: Error finding container ece6e6b8b09727be58549ee097761db91e89799b2434d3ca3b9f2e0baf72f545: Status 404 returned error can't find the container with id ece6e6b8b09727be58549ee097761db91e89799b2434d3ca3b9f2e0baf72f545
Jul 07 17:00:17.955709 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:17.955675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" event={"ID":"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa","Type":"ContainerStarted","Data":"fa4b58fda983559f36f9bc1bf22ab6d2acaeb85dc608b21a6487d492baa9787a"}
Jul 07 17:00:17.956060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:17.955715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" event={"ID":"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa","Type":"ContainerStarted","Data":"ece6e6b8b09727be58549ee097761db91e89799b2434d3ca3b9f2e0baf72f545"}
Jul 07 17:00:17.981268 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:17.981224    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" podStartSLOduration=1.9812099010000002 podStartE2EDuration="1.981209901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:00:17.980287986 +0000 UTC m=+2216.513204628" watchObservedRunningTime="2026-07-07 17:00:17.981209901 +0000 UTC m=+2216.514126543"
Jul 07 17:00:27.005673 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:27.005634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:27.006237 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:27.005707    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:27.021321 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:27.021300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:28.022324 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:28.022296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:00:54.046538 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:00:54.046447    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:01:09.065830 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.065772    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" podUID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" containerName="main" containerID="cri-o://fa4b58fda983559f36f9bc1bf22ab6d2acaeb85dc608b21a6487d492baa9787a" gracePeriod=45
Jul 07 17:01:09.228303 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.228273    2565 generic.go:356] "Generic (PLEG): container finished" podID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" containerID="fa4b58fda983559f36f9bc1bf22ab6d2acaeb85dc608b21a6487d492baa9787a" exitCode=0
Jul 07 17:01:09.228438 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.228351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" event={"ID":"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa","Type":"ContainerDied","Data":"fa4b58fda983559f36f9bc1bf22ab6d2acaeb85dc608b21a6487d492baa9787a"}
Jul 07 17:01:09.306280 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.306258    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:01:09.434476 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434476 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434442    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434676 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434493    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434676 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bmqf\" (UniqueName: \"kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434676 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434569    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434937 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434686    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs\") pod \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\" (UID: \"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa\") "
Jul 07 17:01:09.434937 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434729    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home" (OuterVolumeSpecName: "home") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:09.434937 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434779    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:09.434937 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.434813    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:09.435135 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.435062    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:09.435135 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.435076    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:09.435135 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.435085    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:09.436672 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.436644    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm" (OuterVolumeSpecName: "dshm") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:09.436777 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.436675    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf" (OuterVolumeSpecName: "kube-api-access-5bmqf") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "kube-api-access-5bmqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:01:09.436777 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.436699    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" (UID: "4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:01:09.535710 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.535685    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmqf\" (UniqueName: \"kubernetes.io/projected/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-kube-api-access-5bmqf\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:09.535710 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.535706    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:09.535876 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:09.535716    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:10.234114 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:10.234087    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"
Jul 07 17:01:10.234114 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:10.234097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb" event={"ID":"4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa","Type":"ContainerDied","Data":"ece6e6b8b09727be58549ee097761db91e89799b2434d3ca3b9f2e0baf72f545"}
Jul 07 17:01:10.234476 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:10.234151    2565 scope.go:117] "RemoveContainer" containerID="fa4b58fda983559f36f9bc1bf22ab6d2acaeb85dc608b21a6487d492baa9787a"
Jul 07 17:01:10.255805 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:10.255765    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:01:10.260140 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:10.260121    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-reuse-37478f/test-model-37478f-kserve-dc8df9df9-fsjxb"]
Jul 07 17:01:12.096055 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:12.096023    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" path="/var/lib/kubelet/pods/4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa/volumes"
Jul 07 17:01:53.742911 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:53.742875    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 17:01:55.722761 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:55.722727    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 17:01:55.723138 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:55.722985    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" podUID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" containerName="maas-api" containerID="cri-o://01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993" gracePeriod=30
Jul 07 17:01:55.967120 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:55.967094    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 17:01:56.118147 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.118118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls\") pod \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") "
Jul 07 17:01:56.118312 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.118161    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6v86\" (UniqueName: \"kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86\") pod \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") "
Jul 07 17:01:56.118312 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.118218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca\") pod \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\" (UID: \"6ea67efc-4e3c-491f-b9e5-719e172af3fc\") "
Jul 07 17:01:56.118619 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.118584    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ea67efc-4e3c-491f-b9e5-719e172af3fc" (UID: "6ea67efc-4e3c-491f-b9e5-719e172af3fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:01:56.120192 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.120165    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86" (OuterVolumeSpecName: "kube-api-access-b6v86") pod "6ea67efc-4e3c-491f-b9e5-719e172af3fc" (UID: "6ea67efc-4e3c-491f-b9e5-719e172af3fc"). InnerVolumeSpecName "kube-api-access-b6v86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:01:56.120280 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.120165    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ea67efc-4e3c-491f-b9e5-719e172af3fc" (UID: "6ea67efc-4e3c-491f-b9e5-719e172af3fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:01:56.219312 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.219289    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ea67efc-4e3c-491f-b9e5-719e172af3fc-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:56.219312 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.219315    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v86\" (UniqueName: \"kubernetes.io/projected/6ea67efc-4e3c-491f-b9e5-719e172af3fc-kube-api-access-b6v86\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:56.219312 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.219325    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea67efc-4e3c-491f-b9e5-719e172af3fc-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:01:56.470596 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.470535    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" containerID="01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993" exitCode=0
Jul 07 17:01:56.470596 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.470579    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"
Jul 07 17:01:56.470726 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.470599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" event={"ID":"6ea67efc-4e3c-491f-b9e5-719e172af3fc","Type":"ContainerDied","Data":"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"}
Jul 07 17:01:56.470726 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.470640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst" event={"ID":"6ea67efc-4e3c-491f-b9e5-719e172af3fc","Type":"ContainerDied","Data":"04c6f16bca96be13c4d86570fab5f56c8732e84ead92c195f4e10fb951cd0897"}
Jul 07 17:01:56.470726 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.470663    2565 scope.go:117] "RemoveContainer" containerID="01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"
Jul 07 17:01:56.481564 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.481548    2565 scope.go:117] "RemoveContainer" containerID="01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"
Jul 07 17:01:56.481848 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:01:56.481829    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993\": container with ID starting with 01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993 not found: ID does not exist" containerID="01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"
Jul 07 17:01:56.481907 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.481857    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993"} err="failed to get container status \"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993\": rpc error: code = NotFound desc = could not find container \"01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993\": container with ID starting with 01961185d9a502a6d79676a80ae7258cb37ad2fb74d2f6ea2cb0fa421d8c7993 not found: ID does not exist"
Jul 07 17:01:56.498759 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.498739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 17:01:56.504697 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:56.504677    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1b64ca-65f4cc7fc7-zzvst"]
Jul 07 17:01:58.096026 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:01:58.095994    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" path="/var/lib/kubelet/pods/6ea67efc-4e3c-491f-b9e5-719e172af3fc/volumes"
Jul 07 17:02:08.761635 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:08.761593    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" podUID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" containerName="main" containerID="cri-o://f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d" gracePeriod=45
Jul 07 17:02:09.010152 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.010125    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 17:02:09.120770 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.120905 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.120905 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120823    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.120905 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120858    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.121018 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgvhf\" (UniqueName: \"kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.121018 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.120927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home\") pod \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\" (UID: \"9fb305a0-6dc3-4599-9ecf-db9538e6832e\") "
Jul 07 17:02:09.121113 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.121042    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:09.121263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.121238    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.121263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.121211    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:09.121374 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.121269    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home" (OuterVolumeSpecName: "home") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:09.122878 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.122845    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm" (OuterVolumeSpecName: "dshm") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:09.122878 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.122859    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf" (OuterVolumeSpecName: "kube-api-access-vgvhf") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "kube-api-access-vgvhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:02:09.123000 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.122886    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fb305a0-6dc3-4599-9ecf-db9538e6832e" (UID: "9fb305a0-6dc3-4599-9ecf-db9538e6832e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:02:09.222262 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.222244    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.222262 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.222262    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb305a0-6dc3-4599-9ecf-db9538e6832e-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.222402 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.222271    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.222402 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.222279    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgvhf\" (UniqueName: \"kubernetes.io/projected/9fb305a0-6dc3-4599-9ecf-db9538e6832e-kube-api-access-vgvhf\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.222402 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.222286    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb305a0-6dc3-4599-9ecf-db9538e6832e-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:09.542646 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.542578    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" containerID="f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d" exitCode=0
Jul 07 17:02:09.542646 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.542622    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"
Jul 07 17:02:09.542819 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.542661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" event={"ID":"9fb305a0-6dc3-4599-9ecf-db9538e6832e","Type":"ContainerDied","Data":"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"}
Jul 07 17:02:09.542819 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.542706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr" event={"ID":"9fb305a0-6dc3-4599-9ecf-db9538e6832e","Type":"ContainerDied","Data":"08a5c6c517380d131851dccb70011891f1f1c4abddb58ff621007acd5677144a"}
Jul 07 17:02:09.542819 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.542724    2565 scope.go:117] "RemoveContainer" containerID="f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"
Jul 07 17:02:09.551438 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.551420    2565 scope.go:117] "RemoveContainer" containerID="f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"
Jul 07 17:02:09.551705 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:09.551686    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d\": container with ID starting with f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d not found: ID does not exist" containerID="f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"
Jul 07 17:02:09.551765 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.551716    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d"} err="failed to get container status \"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d\": rpc error: code = NotFound desc = could not find container \"f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d\": container with ID starting with f0ce8c9c5bab91c9a7c08dbad691843c0520b9b6bcf6506e57289505cd9ab35d not found: ID does not exist"
Jul 07 17:02:09.564610 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.564592    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 17:02:09.568305 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:09.568287    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1b64ca/test-model-1b64ca-kserve-57b4d688b7-xs6kr"]
Jul 07 17:02:10.096137 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:10.096099    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" path="/var/lib/kubelet/pods/9fb305a0-6dc3-4599-9ecf-db9538e6832e/volumes"
Jul 07 17:02:37.862862 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:37.862827    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 17:02:37.863342 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:37.863078    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" containerID="cri-o://b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552" gracePeriod=30
Jul 07 17:02:39.163045 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.163017    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 17:02:39.163459 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.163184    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" podUID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" containerName="maas-api" containerID="cri-o://3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6" gracePeriod=30
Jul 07 17:02:39.409362 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.409334    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 17:02:39.460169 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.460098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca\") pod \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") "
Jul 07 17:02:39.460296 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.460183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") pod \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") "
Jul 07 17:02:39.460296 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.460273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm4w6\" (UniqueName: \"kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6\") pod \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\" (UID: \"3b902c4c-7720-4a72-ac58-3beb2569ec7a\") "
Jul 07 17:02:39.460420 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.460397    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b902c4c-7720-4a72-ac58-3beb2569ec7a" (UID: "3b902c4c-7720-4a72-ac58-3beb2569ec7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:02:39.460563 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.460548    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b902c4c-7720-4a72-ac58-3beb2569ec7a-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:39.462099 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.462077    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b902c4c-7720-4a72-ac58-3beb2569ec7a" (UID: "3b902c4c-7720-4a72-ac58-3beb2569ec7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:02:39.462212 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.462135    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6" (OuterVolumeSpecName: "kube-api-access-mm4w6") pod "3b902c4c-7720-4a72-ac58-3beb2569ec7a" (UID: "3b902c4c-7720-4a72-ac58-3beb2569ec7a"). InnerVolumeSpecName "kube-api-access-mm4w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:02:39.561473 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.561452    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b902c4c-7720-4a72-ac58-3beb2569ec7a-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:39.561473 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.561471    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4w6\" (UniqueName: \"kubernetes.io/projected/3b902c4c-7720-4a72-ac58-3beb2569ec7a-kube-api-access-mm4w6\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:39.700687 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.700659    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" containerID="3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6" exitCode=0
Jul 07 17:02:39.700793 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.700708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" event={"ID":"3b902c4c-7720-4a72-ac58-3beb2569ec7a","Type":"ContainerDied","Data":"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"}
Jul 07 17:02:39.700793 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.700718    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"
Jul 07 17:02:39.700793 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.700748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv" event={"ID":"3b902c4c-7720-4a72-ac58-3beb2569ec7a","Type":"ContainerDied","Data":"6fde5b9eb0e8043ff9e84b2c50904033297030f3475cb51fc29ce75c9b996726"}
Jul 07 17:02:39.700793 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.700769    2565 scope.go:117] "RemoveContainer" containerID="3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"
Jul 07 17:02:39.710574 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.710545    2565 scope.go:117] "RemoveContainer" containerID="3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"
Jul 07 17:02:39.710845 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:39.710825    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6\": container with ID starting with 3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6 not found: ID does not exist" containerID="3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"
Jul 07 17:02:39.710912 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.710854    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6"} err="failed to get container status \"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6\": rpc error: code = NotFound desc = could not find container \"3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6\": container with ID starting with 3fbbbffac00281977f91fceb295e72d2918b99c521f4875cf056796d01a3ccd6 not found: ID does not exist"
Jul 07 17:02:39.723542 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.723517    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 17:02:39.727363 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.727346    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-4b5d6a-54df947657-98lvv"]
Jul 07 17:02:39.813942 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.813918    2565 patch_prober.go:28] interesting pod/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 17:02:39.814041 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:39.813965    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:02:40.096178 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:40.096155    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" path="/var/lib/kubelet/pods/3b902c4c-7720-4a72-ac58-3beb2569ec7a/volumes"
Jul 07 17:02:43.114007 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.113982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 17:02:43.189979 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.189918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.189979 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.189956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.189979 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.189977    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190153    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h84p2\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190223 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token\") pod \"ca2d106b-3af4-429c-b78e-6a6794085fdb\" (UID: \"ca2d106b-3af4-429c-b78e-6a6794085fdb\") "
Jul 07 17:02:43.190525 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190254    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:43.190525 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190268    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:43.190525 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190298    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:02:43.190525 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190475    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:43.190525 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190511    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2d106b-3af4-429c-b78e-6a6794085fdb-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.190772 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190530    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.190772 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.190543    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.191057 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.191026    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:43.192092 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.192070    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:43.192197 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.192115    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 17:02:43.192258 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.192217    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:02:43.192452 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.192438    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2" (OuterVolumeSpecName: "kube-api-access-h84p2") pod "ca2d106b-3af4-429c-b78e-6a6794085fdb" (UID: "ca2d106b-3af4-429c-b78e-6a6794085fdb"). InnerVolumeSpecName "kube-api-access-h84p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:02:43.291917 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291889    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.291917 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291914    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84p2\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-kube-api-access-h84p2\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.292049 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291929    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.292049 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291940    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.292049 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291951    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.292049 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.291961    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2d106b-3af4-429c-b78e-6a6794085fdb-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:02:43.722046 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.722015    2565 generic.go:356] "Generic (PLEG): container finished" podID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerID="b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552" exitCode=0
Jul 07 17:02:43.722177 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.722062    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"
Jul 07 17:02:43.722177 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.722108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerDied","Data":"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"}
Jul 07 17:02:43.722177 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.722171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v" event={"ID":"ca2d106b-3af4-429c-b78e-6a6794085fdb","Type":"ContainerDied","Data":"076095c39a5caae62d0af21ea583419f6b30e0712782843466410f424494c077"}
Jul 07 17:02:43.722310 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.722199    2565 scope.go:117] "RemoveContainer" containerID="b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"
Jul 07 17:02:43.731193 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.731178    2565 scope.go:117] "RemoveContainer" containerID="bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"
Jul 07 17:02:43.740631 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.740614    2565 scope.go:117] "RemoveContainer" containerID="b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"
Jul 07 17:02:43.740958 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:43.740927    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552\": container with ID starting with b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552 not found: ID does not exist" containerID="b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"
Jul 07 17:02:43.741068 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.740963    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552"} err="failed to get container status \"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552\": rpc error: code = NotFound desc = could not find container \"b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552\": container with ID starting with b36c8c4c9d5189cbd71bdeb76b4e6244b2450356e52bebac50a84b53f429f552 not found: ID does not exist"
Jul 07 17:02:43.741068 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.740986    2565 scope.go:117] "RemoveContainer" containerID="bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"
Jul 07 17:02:43.741408 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:43.741391    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443\": container with ID starting with bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443 not found: ID does not exist" containerID="bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"
Jul 07 17:02:43.741467 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.741413    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443"} err="failed to get container status \"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443\": rpc error: code = NotFound desc = could not find container \"bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443\": container with ID starting with bb2559aaa5ebbeb04cd2a699864d0094a7b9101b04108739f19177469e07b443 not found: ID does not exist"
Jul 07 17:02:43.746622 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.746603    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 17:02:43.751995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:43.751976    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1b64ca-openshift-default-dd79997d7-x7j5v"]
Jul 07 17:02:44.096692 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:44.096665    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" path="/var/lib/kubelet/pods/ca2d106b-3af4-429c-b78e-6a6794085fdb/volumes"
Jul 07 17:02:46.123689 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.123660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124023    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" containerName="maas-api"
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124035    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" containerName="maas-api"
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124043    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124048    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124058    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" containerName="maas-api"
Jul 07 17:02:46.124060 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" containerName="maas-api"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124079    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124091    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124106    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124113    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124182    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2e5a3f-f19e-4d6b-99d5-5c86e20b53fa" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124192    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124200    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb305a0-6dc3-4599-9ecf-db9538e6832e" containerName="main"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124205    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea67efc-4e3c-491f-b9e5-719e172af3fc" containerName="maas-api"
Jul 07 17:02:46.124263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.124210    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b902c4c-7720-4a72-ac58-3beb2569ec7a" containerName="maas-api"
Jul 07 17:02:46.128759 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.128738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.131623 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.131584    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-966483-openshift-default-dockercfg-fxdq2\""
Jul 07 17:02:46.147821 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.147803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:02:46.213269 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213363 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213363 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213363 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lv4\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213498 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213498 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213567 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213635 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.213674 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.213652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.314641 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.314641 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.314877 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.314877 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.314975 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315026 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.314969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lv4\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315026 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315012    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315134 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315134 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315134 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315134 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315097    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315398 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315372    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315498 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315375    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.315498 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.315398    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.317519 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.317500    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.317835 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.317815    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.326767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.326741    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lv4\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.326867 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.326776    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token\") pod \"e2e-api-a-966483-openshift-default-55c467bb59-nvthk\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") " pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.438317 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.438264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:46.576638 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.576602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:02:46.579966 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:02:46.579943    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad59e515_890c_4448_8912_a92ac01c092f.slice/crio-a844d5e5a6528df30be9c877144b6632cfd6f7283832ba40f06fbaed826db327 WatchSource:0}: Error finding container a844d5e5a6528df30be9c877144b6632cfd6f7283832ba40f06fbaed826db327: Status 404 returned error can't find the container with id a844d5e5a6528df30be9c877144b6632cfd6f7283832ba40f06fbaed826db327
Jul 07 17:02:46.582021 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.581989    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:02:46.582119 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.582068    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:02:46.582174 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.582115    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:02:46.738935 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.738857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerStarted","Data":"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"}
Jul 07 17:02:46.738935 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.738914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerStarted","Data":"a844d5e5a6528df30be9c877144b6632cfd6f7283832ba40f06fbaed826db327"}
Jul 07 17:02:46.770314 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:46.770268    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podStartSLOduration=0.770251309 podStartE2EDuration="770.251309ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:02:46.769436201 +0000 UTC m=+2365.302352934" watchObservedRunningTime="2026-07-07 17:02:46.770251309 +0000 UTC m=+2365.303167949"
Jul 07 17:02:47.438952 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:47.438919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:47.443289 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:47.443266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:47.743641 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:47.743569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:47.744365 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:47.744344    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:02:57.843209 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.843177    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:02:57.843635 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.843594    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2d106b-3af4-429c-b78e-6a6794085fdb" containerName="istio-proxy"
Jul 07 17:02:57.846637 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.846618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:57.849287 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.849264    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-4f1f9c-serving-cert\""
Jul 07 17:02:57.849486 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.849278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\""
Jul 07 17:02:57.873957 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.873937    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:02:57.915805 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.915180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:57.915805 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.915219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:57.915805 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:57.915310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq476\" (UniqueName: \"kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.015873 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.015837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.016226 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.016206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.017153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.017133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq476\" (UniqueName: \"kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.017608 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.016983    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.017767 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:58.016156    2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-4f1f9c-serving-cert: secret "maas-api-e2e-api-b-4f1f9c-serving-cert" not found
Jul 07 17:02:58.017945 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:02:58.017935    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls podName:01fbd722-d2b5-4c48-8379-8772e577601c nodeName:}" failed. No retries permitted until 2026-07-07 17:02:58.517914118 +0000 UTC m=+2377.050830745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls") pod "maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" (UID: "01fbd722-d2b5-4c48-8379-8772e577601c") : secret "maas-api-e2e-api-b-4f1f9c-serving-cert" not found
Jul 07 17:02:58.033750 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.033719    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq476\" (UniqueName: \"kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.522967 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.522886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.525170 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.525150    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") pod \"maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") " pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.759157 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.759119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:58.913282 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:58.913241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:02:58.916615 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:02:58.916587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fbd722_d2b5_4c48_8379_8772e577601c.slice/crio-d5c107dc06916399d02d081bac5204bdc1a251ef5c4a3536020104517e023d6f WatchSource:0}: Error finding container d5c107dc06916399d02d081bac5204bdc1a251ef5c4a3536020104517e023d6f: Status 404 returned error can't find the container with id d5c107dc06916399d02d081bac5204bdc1a251ef5c4a3536020104517e023d6f
Jul 07 17:02:59.806185 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:59.806093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" event={"ID":"01fbd722-d2b5-4c48-8379-8772e577601c","Type":"ContainerStarted","Data":"22472adba12f10207390eeda052d262dbab85c3d471a8bc6d462f7dbac7409b0"}
Jul 07 17:02:59.806185 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:59.806173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:02:59.806374 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:02:59.806200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" event={"ID":"01fbd722-d2b5-4c48-8379-8772e577601c","Type":"ContainerStarted","Data":"d5c107dc06916399d02d081bac5204bdc1a251ef5c4a3536020104517e023d6f"}
Jul 07 17:03:01.819398 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.819367    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad59e515-890c-4448-8912-a92ac01c092f" containerID="f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da" exitCode=0
Jul 07 17:03:01.819868 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.819446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerDied","Data":"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"}
Jul 07 17:03:01.819868 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.819840    2565 scope.go:117] "RemoveContainer" containerID="f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"
Jul 07 17:03:01.820862 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.820832    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:01.821016 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.820920    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:01.821016 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.820988    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:01.852479 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:01.852433    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" podStartSLOduration=4.436862204 podStartE2EDuration="4.852418593s" totalImagesPullingTime="415.556389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:02:59.83141377 +0000 UTC m=+2378.364330431" watchObservedRunningTime="2026-07-07 17:03:01.852418593 +0000 UTC m=+2380.385335235"
Jul 07 17:03:02.744449 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:02.744411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:02.832887 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:02.832851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerStarted","Data":"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"}
Jul 07 17:03:03.438943 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:03.438904    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:03.439741 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:03.439719    2565 patch_prober.go:28] interesting pod/e2e-api-a-966483-openshift-default-55c467bb59-nvthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 07 17:03:03.439814 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:03.439769    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 07 17:03:04.438877 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:04.438841    2565 patch_prober.go:28] interesting pod/e2e-api-a-966483-openshift-default-55c467bb59-nvthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 07 17:03:04.439221 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:04.438892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 07 17:03:05.438506 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:05.438478    2565 patch_prober.go:28] interesting pod/e2e-api-a-966483-openshift-default-55c467bb59-nvthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 07 17:03:05.438668 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:05.438537    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 07 17:03:05.823970 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:05.823935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:03:06.439674 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:06.439638    2565 patch_prober.go:28] interesting pod/e2e-api-a-966483-openshift-default-55c467bb59-nvthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 07 17:03:06.439850 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:06.439730    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 07 17:03:07.442122 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:07.442095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:07.442565 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:07.442442    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:07.443187 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:07.443168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:17.442119 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:17.442086    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:03:17.442646 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:17.442307    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" containerID="cri-o://70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29" gracePeriod=30
Jul 07 17:03:18.734955 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.734925    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:03:18.736851 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.736305    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" podUID="01fbd722-d2b5-4c48-8379-8772e577601c" containerName="maas-api" containerID="cri-o://22472adba12f10207390eeda052d262dbab85c3d471a8bc6d462f7dbac7409b0" gracePeriod=30
Jul 07 17:03:18.921108 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.921076    2565 generic.go:356] "Generic (PLEG): container finished" podID="01fbd722-d2b5-4c48-8379-8772e577601c" containerID="22472adba12f10207390eeda052d262dbab85c3d471a8bc6d462f7dbac7409b0" exitCode=0
Jul 07 17:03:18.921247 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.921153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" event={"ID":"01fbd722-d2b5-4c48-8379-8772e577601c","Type":"ContainerDied","Data":"22472adba12f10207390eeda052d262dbab85c3d471a8bc6d462f7dbac7409b0"}
Jul 07 17:03:18.977638 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.977620    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:03:18.997888 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.997841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca\") pod \"01fbd722-d2b5-4c48-8379-8772e577601c\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") "
Jul 07 17:03:18.997989 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.997944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq476\" (UniqueName: \"kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476\") pod \"01fbd722-d2b5-4c48-8379-8772e577601c\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") "
Jul 07 17:03:18.997989 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.997973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") pod \"01fbd722-d2b5-4c48-8379-8772e577601c\" (UID: \"01fbd722-d2b5-4c48-8379-8772e577601c\") "
Jul 07 17:03:18.998165 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.998144    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01fbd722-d2b5-4c48-8379-8772e577601c" (UID: "01fbd722-d2b5-4c48-8379-8772e577601c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:18.998242 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.998220    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fbd722-d2b5-4c48-8379-8772e577601c-openshift-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:18.999874 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.999852    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476" (OuterVolumeSpecName: "kube-api-access-kq476") pod "01fbd722-d2b5-4c48-8379-8772e577601c" (UID: "01fbd722-d2b5-4c48-8379-8772e577601c"). InnerVolumeSpecName "kube-api-access-kq476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:03:18.999951 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:18.999934    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01fbd722-d2b5-4c48-8379-8772e577601c" (UID: "01fbd722-d2b5-4c48-8379-8772e577601c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:03:19.099355 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.099332    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq476\" (UniqueName: \"kubernetes.io/projected/01fbd722-d2b5-4c48-8379-8772e577601c-kube-api-access-kq476\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:19.099355 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.099354    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fbd722-d2b5-4c48-8379-8772e577601c-maas-api-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:19.933248 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.933216    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"
Jul 07 17:03:19.933248 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.933234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995" event={"ID":"01fbd722-d2b5-4c48-8379-8772e577601c","Type":"ContainerDied","Data":"d5c107dc06916399d02d081bac5204bdc1a251ef5c4a3536020104517e023d6f"}
Jul 07 17:03:19.933712 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.933317    2565 scope.go:117] "RemoveContainer" containerID="22472adba12f10207390eeda052d262dbab85c3d471a8bc6d462f7dbac7409b0"
Jul 07 17:03:19.956999 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.956977    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:03:19.964880 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:19.964857    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-4f1f9c-644f64bb6c-s9995"]
Jul 07 17:03:20.096149 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:20.096120    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fbd722-d2b5-4c48-8379-8772e577601c" path="/var/lib/kubelet/pods/01fbd722-d2b5-4c48-8379-8772e577601c/volumes"
Jul 07 17:03:22.444602 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.444538    2565 patch_prober.go:28] interesting pod/e2e-api-a-966483-openshift-default-55c467bb59-nvthk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 17:03:22.444963 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.444595    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:03:22.691573 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.691551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:22.730109 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730109 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730109 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730301    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9lv4\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730345 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730338    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert\") pod \"ad59e515-890c-4448-8912-a92ac01c092f\" (UID: \"ad59e515-890c-4448-8912-a92ac01c092f\") "
Jul 07 17:03:22.730642 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730361    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:22.730642 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730524    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:22.730749 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730683    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.730749 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730704    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.730874 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730773    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:22.731218 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.730980    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:22.731729 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.731700    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:22.732462 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.732420    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:03:22.732559 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.732468    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 17:03:22.732617 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.732575    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4" (OuterVolumeSpecName: "kube-api-access-c9lv4") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "kube-api-access-c9lv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:03:22.732617 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.732600    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad59e515-890c-4448-8912-a92ac01c092f" (UID: "ad59e515-890c-4448-8912-a92ac01c092f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:22.831648 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831626    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lv4\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-kube-api-access-c9lv4\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831648 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831646    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad59e515-890c-4448-8912-a92ac01c092f-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831655    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831663    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad59e515-890c-4448-8912-a92ac01c092f-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831670    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad59e515-890c-4448-8912-a92ac01c092f-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831677    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.831767 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.831684    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad59e515-890c-4448-8912-a92ac01c092f-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:03:22.952181 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.952154    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad59e515-890c-4448-8912-a92ac01c092f" containerID="70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29" exitCode=0
Jul 07 17:03:22.952266 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.952207    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"
Jul 07 17:03:22.952266 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.952235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerDied","Data":"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"}
Jul 07 17:03:22.952335 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.952284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk" event={"ID":"ad59e515-890c-4448-8912-a92ac01c092f","Type":"ContainerDied","Data":"a844d5e5a6528df30be9c877144b6632cfd6f7283832ba40f06fbaed826db327"}
Jul 07 17:03:22.952335 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.952309    2565 scope.go:117] "RemoveContainer" containerID="70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"
Jul 07 17:03:22.960613 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.960597    2565 scope.go:117] "RemoveContainer" containerID="f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"
Jul 07 17:03:22.967838 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.967819    2565 scope.go:117] "RemoveContainer" containerID="70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"
Jul 07 17:03:22.968074 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:03:22.968058    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29\": container with ID starting with 70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29 not found: ID does not exist" containerID="70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"
Jul 07 17:03:22.968120 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.968080    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29"} err="failed to get container status \"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29\": rpc error: code = NotFound desc = could not find container \"70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29\": container with ID starting with 70c32d4ce092e88e1bd600a127c8b07367cb265ee062f75cab218a7aa7fbde29 not found: ID does not exist"
Jul 07 17:03:22.968120 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.968094    2565 scope.go:117] "RemoveContainer" containerID="f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"
Jul 07 17:03:22.968319 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:03:22.968303    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da\": container with ID starting with f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da not found: ID does not exist" containerID="f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"
Jul 07 17:03:22.968366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.968322    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da"} err="failed to get container status \"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da\": rpc error: code = NotFound desc = could not find container \"f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da\": container with ID starting with f48708ab443d1dfe1debf4a3cf64a71e1c93fd013e9e2d90ffb8f58e431246da not found: ID does not exist"
Jul 07 17:03:22.977812 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.976674    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:03:22.983216 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:22.983162    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-966483-openshift-default-55c467bb59-nvthk"]
Jul 07 17:03:24.095704 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:24.095672    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad59e515-890c-4448-8912-a92ac01c092f" path="/var/lib/kubelet/pods/ad59e515-890c-4448-8912-a92ac01c092f/volumes"
Jul 07 17:03:25.706800 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.706749    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707155    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fbd722-d2b5-4c48-8379-8772e577601c" containerName="maas-api"
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707167    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fbd722-d2b5-4c48-8379-8772e577601c" containerName="maas-api"
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707182    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707209    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.707229 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707215    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.707511 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707273    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.707511 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707279    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fbd722-d2b5-4c48-8379-8772e577601c" containerName="maas-api"
Jul 07 17:03:25.707511 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.707284    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad59e515-890c-4448-8912-a92ac01c092f" containerName="istio-proxy"
Jul 07 17:03:25.711119 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.711099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.713717 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.713698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-58e7b2-openshift-default-dockercfg-c2bq5\""
Jul 07 17:03:25.734759 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.734741    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:03:25.751635 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751761 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751761 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.751995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.751921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtrn\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.752200 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.752029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.752200 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.752050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852686 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.852889 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtrn\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853191 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853191 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.852955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853191 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.853168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853329 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.853230    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853459 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.853438    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853511 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.853497    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.853561 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.853514    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.854907 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.854891    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.855128 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.855111    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.863017 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.862997    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtrn\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:25.863102 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:25.863061    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token\") pod \"e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") " pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:26.020460 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.020391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:26.158232 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.158209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:03:26.158888 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:03:26.158868    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda588edf8_0338_4391_9db3_552502f60506.slice/crio-7ba597ac5d30559cb691bff60100bc9715f591be5d86ae44ba6584891e12e46f WatchSource:0}: Error finding container 7ba597ac5d30559cb691bff60100bc9715f591be5d86ae44ba6584891e12e46f: Status 404 returned error can't find the container with id 7ba597ac5d30559cb691bff60100bc9715f591be5d86ae44ba6584891e12e46f
Jul 07 17:03:26.160754 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.160728    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:26.160824 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.160808    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:26.160873 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.160862    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:26.974050 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.974014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerStarted","Data":"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"}
Jul 07 17:03:26.974423 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:26.974062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerStarted","Data":"7ba597ac5d30559cb691bff60100bc9715f591be5d86ae44ba6584891e12e46f"}
Jul 07 17:03:27.008039 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:27.007986    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" podStartSLOduration=2.007968067 podStartE2EDuration="2.007968067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:03:27.006815489 +0000 UTC m=+2405.539732153" watchObservedRunningTime="2026-07-07 17:03:27.007968067 +0000 UTC m=+2405.540884709"
Jul 07 17:03:27.020791 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:27.020766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:27.025690 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:27.025673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:27.979921 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:27.979889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:27.980899 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:27.980880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:34.140806 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.140198    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:03:34.144863 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.144834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.147762 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.147736    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-fa4616-openshift-default-dockercfg-4h9n2\""
Jul 07 17:03:34.164146 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.164123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkldq\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.232809 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.229538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331057 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331251 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331251 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkldq\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331251 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331251 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331480 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331480 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331480 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331480 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.331897 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.331864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.332478 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.332116    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.332478 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.332155    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.332625 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.332477    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.332883 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.332858    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.334638 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.334318    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.334638 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.334556    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.341986 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.341963    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.344725 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.344706    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkldq\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq\") pod \"e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") " pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.456528 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.456458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:34.628668 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.628627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:03:34.638461 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:03:34.638431    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863e23a6_d873_478e_bae2_806125b70ec1.slice/crio-5a3137447e2dca6bbfda5596e8a1356762a6d9e273b17d34d43e4e3b166f6172 WatchSource:0}: Error finding container 5a3137447e2dca6bbfda5596e8a1356762a6d9e273b17d34d43e4e3b166f6172: Status 404 returned error can't find the container with id 5a3137447e2dca6bbfda5596e8a1356762a6d9e273b17d34d43e4e3b166f6172
Jul 07 17:03:34.640457 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.640426    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:34.640569 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.640517    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:34.640627 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:34.640584    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:35.020472 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.020423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerStarted","Data":"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"}
Jul 07 17:03:35.020661 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.020536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerStarted","Data":"5a3137447e2dca6bbfda5596e8a1356762a6d9e273b17d34d43e4e3b166f6172"}
Jul 07 17:03:35.047489 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.047435    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podStartSLOduration=1.047417323 podStartE2EDuration="1.047417323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:03:35.046071661 +0000 UTC m=+2413.578988303" watchObservedRunningTime="2026-07-07 17:03:35.047417323 +0000 UTC m=+2413.580333966"
Jul 07 17:03:35.457348 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.457320    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:35.458811 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.458776    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 07 17:03:35.458957 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:35.458845    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 17:03:36.461086 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:36.461055    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:37.036934 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:37.036902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:37.039919 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:37.039899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:41.057139 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.057107    2565 generic.go:356] "Generic (PLEG): container finished" podID="a588edf8-0338-4391-9db3-552502f60506" containerID="f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35" exitCode=0
Jul 07 17:03:41.057506 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.057146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerDied","Data":"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"}
Jul 07 17:03:41.057506 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.057502    2565 scope.go:117] "RemoveContainer" containerID="f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"
Jul 07 17:03:41.058434 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.058405    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:41.058503 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.058475    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:41.058538 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:41.058507    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:42.064400 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:42.064359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerStarted","Data":"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"}
Jul 07 17:03:42.979838 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:42.979801    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:43.021180 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:43.021151    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:43.022402 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:43.022377    2565 patch_prober.go:28] interesting pod/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 07 17:03:43.022607 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:43.022581    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 07 17:03:44.020924 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:44.020888    2565 patch_prober.go:28] interesting pod/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 07 17:03:44.021470 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:44.021436    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 07 17:03:45.021744 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:45.021707    2565 patch_prober.go:28] interesting pod/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 07 17:03:45.022366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:45.022333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 07 17:03:46.024995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:46.024967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:46.025881 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:46.025863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:03:53.077914 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:53.077881    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 07 17:03:53.078334 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:53.077931    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 17:03:54.128678 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.128649    2565 generic.go:356] "Generic (PLEG): container finished" podID="863e23a6-d873-478e-bae2-806125b70ec1" containerID="de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c" exitCode=0
Jul 07 17:03:54.129080 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.128720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerDied","Data":"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"}
Jul 07 17:03:54.129123 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.129092    2565 scope.go:117] "RemoveContainer" containerID="de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"
Jul 07 17:03:54.129997 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.129975    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:54.130068 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.130039    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:54.130106 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:54.130095    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:03:55.135930 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:55.135894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerStarted","Data":"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"}
Jul 07 17:03:55.457356 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:55.457251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:55.457579 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:55.457550    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 07 17:03:55.457815 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:55.457773    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 17:03:56.456966 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:56.456932    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 07 17:03:56.457358 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:56.456985    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 17:03:57.457612 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:57.457580    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 07 17:03:57.457991 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:57.457632    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 17:03:58.397291 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.397262    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:03:58.399592 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.399575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.403421 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.403398    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"openshift-service-ca.crt\""
Jul 07 17:03:58.403421 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.403414    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"default-dockercfg-fzgk9\""
Jul 07 17:03:58.403590 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.403398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"auth-test-model-58e7b2-kserve-self-signed-certs\""
Jul 07 17:03:58.403590 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.403459    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"kube-root-ca.crt\""
Jul 07 17:03:58.426703 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.426672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:03:58.457035 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.457002    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 07 17:03:58.457190 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.457062    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 17:03:58.553532 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.553532 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.554071 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.554071 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn56\" (UniqueName: \"kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.554071 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.554071 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.553681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.654838 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn56\" (UniqueName: \"kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.654838 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.654838 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655066 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655066 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655066 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.654961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655224 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.655199    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655280 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.655257    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.655335 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.655300    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.657125 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.657105    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.657485 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.657466    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.664488 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.664469    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn56\" (UniqueName: \"kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56\") pod \"auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") " pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.723640 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.723609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:03:58.927543 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:58.927434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:03:58.937334 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:03:58.937299    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ea5a1b_4425_4fb2_ba96_94c7a8e5ecf1.slice/crio-bc69cfb74ba6f7d262dc07749ec6a3261c107af000e1b5f35d22997bab0eed67 WatchSource:0}: Error finding container bc69cfb74ba6f7d262dc07749ec6a3261c107af000e1b5f35d22997bab0eed67: Status 404 returned error can't find the container with id bc69cfb74ba6f7d262dc07749ec6a3261c107af000e1b5f35d22997bab0eed67
Jul 07 17:03:59.156376 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.156329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" event={"ID":"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1","Type":"ContainerStarted","Data":"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"}
Jul 07 17:03:59.156544 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.156384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" event={"ID":"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1","Type":"ContainerStarted","Data":"bc69cfb74ba6f7d262dc07749ec6a3261c107af000e1b5f35d22997bab0eed67"}
Jul 07 17:03:59.181327 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.181244    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" podStartSLOduration=1.181228871 podStartE2EDuration="1.181228871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:03:59.178658979 +0000 UTC m=+2437.711575621" watchObservedRunningTime="2026-07-07 17:03:59.181228871 +0000 UTC m=+2437.714145513"
Jul 07 17:03:59.460366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.460288    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:59.460780 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.460762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:03:59.461446 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:03:59.461427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:04:08.724651 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:04:08.724618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:04:08.725212 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:04:08.724775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:04:08.736356 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:04:08.736334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:04:09.217521 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:04:09.217489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:06:19.118857 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:19.118815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:06:34.139245 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.139175    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" podUID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" containerName="main" containerID="cri-o://8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2" gracePeriod=45
Jul 07 17:06:34.390057 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.389991    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:06:34.472359 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.472728 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472466    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.472728 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472511    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.472728 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.472728 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.472728 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnn56\" (UniqueName: \"kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56\") pod \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\" (UID: \"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1\") "
Jul 07 17:06:34.473107 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472717    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:06:34.473107 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.472969    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home" (OuterVolumeSpecName: "home") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:06:34.473226 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.473097    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:06:34.473226 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.473177    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.473226 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.473202    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.476367 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.476293    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm" (OuterVolumeSpecName: "dshm") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:06:34.476367 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.476293    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:06:34.476707 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.476322    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56" (OuterVolumeSpecName: "kube-api-access-hnn56") pod "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" (UID: "24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1"). InnerVolumeSpecName "kube-api-access-hnn56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:06:34.574061 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.574021    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.574061 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.574053    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.574061 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.574066    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.574304 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.574077    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnn56\" (UniqueName: \"kubernetes.io/projected/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1-kube-api-access-hnn56\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:06:34.931802 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.931752    2565 generic.go:356] "Generic (PLEG): container finished" podID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" containerID="8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2" exitCode=0
Jul 07 17:06:34.932032 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.931834    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"
Jul 07 17:06:34.932032 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.931850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" event={"ID":"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1","Type":"ContainerDied","Data":"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"}
Jul 07 17:06:34.932032 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.931903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb" event={"ID":"24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1","Type":"ContainerDied","Data":"bc69cfb74ba6f7d262dc07749ec6a3261c107af000e1b5f35d22997bab0eed67"}
Jul 07 17:06:34.932032 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.931925    2565 scope.go:117] "RemoveContainer" containerID="8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"
Jul 07 17:06:34.941157 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.941113    2565 scope.go:117] "RemoveContainer" containerID="8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"
Jul 07 17:06:34.941436 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:06:34.941416    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2\": container with ID starting with 8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2 not found: ID does not exist" containerID="8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"
Jul 07 17:06:34.941475 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.941446    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2"} err="failed to get container status \"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2\": rpc error: code = NotFound desc = could not find container \"8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2\": container with ID starting with 8cbaede342613a7ba738a38edeee20836e2869fc71fa4902a005104556c647a2 not found: ID does not exist"
Jul 07 17:06:34.960698 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.960666    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:06:34.970129 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:34.970091    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/auth-test-model-58e7b2-kserve-6ff4d7cc99-49ltb"]
Jul 07 17:06:36.095129 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:06:36.095086    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" path="/var/lib/kubelet/pods/24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1/volumes"
Jul 07 17:07:00.179023 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.178944    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:07:00.179468 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.179335    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" containerName="main"
Jul 07 17:07:00.179468 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.179345    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" containerName="main"
Jul 07 17:07:00.179468 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.179415    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ea5a1b-4425-4fb2-ba96-94c7a8e5ecf1" containerName="main"
Jul 07 17:07:00.181557 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.181540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.194310 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.194290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa4616\"/\"default-dockercfg-jvr5l\""
Jul 07 17:07:00.194427 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.194405    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa4616\"/\"openshift-service-ca.crt\""
Jul 07 17:07:00.194694 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.194678    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa4616\"/\"sub-test-model-fa4616-kserve-self-signed-certs\""
Jul 07 17:07:00.194727 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.194679    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa4616\"/\"kube-root-ca.crt\""
Jul 07 17:07:00.220543 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.220509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:07:00.302026 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.301989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.302228 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.302071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.302228 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.302093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.302228 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.302119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.302228 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.302141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.302390 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.302228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7zq\" (UniqueName: \"kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403583 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403583 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403860 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7zq\" (UniqueName: \"kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403860 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403860 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.403860 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.403847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.404213 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.404190    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.404306 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.404221    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.404306 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.404253    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.406005 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.405980    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.406162 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.406148    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.416037 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.416015    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7zq\" (UniqueName: \"kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq\") pod \"sub-test-model-fa4616-kserve-54775ddf5f-ttntg\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") " pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.490474 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.490375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:00.639866 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:00.639716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:07:00.642566 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:07:00.642528    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c018af9_c75a_4c51_9d34_eb196e265da5.slice/crio-95e56f97dbb2a25de2a87700d8b6d5b13e17af812fd1bc0ac212d71be7f51580 WatchSource:0}: Error finding container 95e56f97dbb2a25de2a87700d8b6d5b13e17af812fd1bc0ac212d71be7f51580: Status 404 returned error can't find the container with id 95e56f97dbb2a25de2a87700d8b6d5b13e17af812fd1bc0ac212d71be7f51580
Jul 07 17:07:01.065396 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:01.065336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" event={"ID":"4c018af9-c75a-4c51-9d34-eb196e265da5","Type":"ContainerStarted","Data":"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"}
Jul 07 17:07:01.065396 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:01.065397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" event={"ID":"4c018af9-c75a-4c51-9d34-eb196e265da5","Type":"ContainerStarted","Data":"95e56f97dbb2a25de2a87700d8b6d5b13e17af812fd1bc0ac212d71be7f51580"}
Jul 07 17:07:01.096945 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:01.096895    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" podStartSLOduration=1.096878537 podStartE2EDuration="1.096878537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:01.095460979 +0000 UTC m=+2619.628377624" watchObservedRunningTime="2026-07-07 17:07:01.096878537 +0000 UTC m=+2619.629795180"
Jul 07 17:07:10.491510 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:10.491465    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:10.491510 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:10.491517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:10.503721 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:10.503695    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:11.127819 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:11.127769    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:43.609350 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:43.609305    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:07:58.628774 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.628734    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" podUID="4c018af9-c75a-4c51-9d34-eb196e265da5" containerName="main" containerID="cri-o://61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e" gracePeriod=45
Jul 07 17:07:58.882094 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.882028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:58.936065 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936250 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936072    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936250 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936200    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936371 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936253    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g7zq\" (UniqueName: \"kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936371 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936371 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936309    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:58.936516 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936384    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir\") pod \"4c018af9-c75a-4c51-9d34-eb196e265da5\" (UID: \"4c018af9-c75a-4c51-9d34-eb196e265da5\") "
Jul 07 17:07:58.936516 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936464    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home" (OuterVolumeSpecName: "home") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:58.936631 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936608    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:58.936724 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936709    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:58.936803 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936729    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:58.936803 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.936743    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:58.938429 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.938406    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:58.938484 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.938412    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq" (OuterVolumeSpecName: "kube-api-access-2g7zq") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "kube-api-access-2g7zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:07:58.938484 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:58.938456    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm" (OuterVolumeSpecName: "dshm") pod "4c018af9-c75a-4c51-9d34-eb196e265da5" (UID: "4c018af9-c75a-4c51-9d34-eb196e265da5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:59.037741 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.037705    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c018af9-c75a-4c51-9d34-eb196e265da5-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:59.037741 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.037739    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7zq\" (UniqueName: \"kubernetes.io/projected/4c018af9-c75a-4c51-9d34-eb196e265da5-kube-api-access-2g7zq\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:59.037741 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.037749    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c018af9-c75a-4c51-9d34-eb196e265da5-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:07:59.357155 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.357115    2565 generic.go:356] "Generic (PLEG): container finished" podID="4c018af9-c75a-4c51-9d34-eb196e265da5" containerID="61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e" exitCode=0
Jul 07 17:07:59.357331 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.357162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"
Jul 07 17:07:59.357331 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.357208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" event={"ID":"4c018af9-c75a-4c51-9d34-eb196e265da5","Type":"ContainerDied","Data":"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"}
Jul 07 17:07:59.357331 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.357262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg" event={"ID":"4c018af9-c75a-4c51-9d34-eb196e265da5","Type":"ContainerDied","Data":"95e56f97dbb2a25de2a87700d8b6d5b13e17af812fd1bc0ac212d71be7f51580"}
Jul 07 17:07:59.357331 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.357281    2565 scope.go:117] "RemoveContainer" containerID="61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"
Jul 07 17:07:59.367351 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.367328    2565 scope.go:117] "RemoveContainer" containerID="61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"
Jul 07 17:07:59.367606 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:07:59.367588    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e\": container with ID starting with 61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e not found: ID does not exist" containerID="61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"
Jul 07 17:07:59.367666 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.367616    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e"} err="failed to get container status \"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e\": rpc error: code = NotFound desc = could not find container \"61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e\": container with ID starting with 61b9a51d0d799bb322c72508e307c05c58052d88786a29d4cb54acda7b8d348e not found: ID does not exist"
Jul 07 17:07:59.383860 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.383830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:07:59.387665 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:07:59.387640    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa4616/sub-test-model-fa4616-kserve-54775ddf5f-ttntg"]
Jul 07 17:08:00.096139 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:00.096104    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c018af9-c75a-4c51-9d34-eb196e265da5" path="/var/lib/kubelet/pods/4c018af9-c75a-4c51-9d34-eb196e265da5/volumes"
Jul 07 17:08:14.792843 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.792699    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:08:14.793620 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.793600    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c018af9-c75a-4c51-9d34-eb196e265da5" containerName="main"
Jul 07 17:08:14.793620 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.793622    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c018af9-c75a-4c51-9d34-eb196e265da5" containerName="main"
Jul 07 17:08:14.793729 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.793721    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c018af9-c75a-4c51-9d34-eb196e265da5" containerName="main"
Jul 07 17:08:14.798411 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.798388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.803845 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.803808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"kube-root-ca.crt\""
Jul 07 17:08:14.805068 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.805047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"default-dockercfg-fzgk9\""
Jul 07 17:08:14.805220 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.805111    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"openshift-service-ca.crt\""
Jul 07 17:08:14.805220 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.805120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"rate-test-model-58e7b2-kserve-self-signed-certs\""
Jul 07 17:08:14.812670 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.812641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:08:14.889766 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.889727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9fb\" (UniqueName: \"kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.889766 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.889769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.890024 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.889874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.890024 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.889929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.890113 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.890027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.890113 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.890062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991088 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9fb\" (UniqueName: \"kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991088 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.991647 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.992036 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991984    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.992036 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.991989    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.992036 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.992020    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.994050 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.994024    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:14.994703 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:14.994678    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:15.003050 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.003021    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9fb\" (UniqueName: \"kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb\") pod \"rate-test-model-58e7b2-kserve-5b47485bd7-87pln\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") " pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:15.108087 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.108045    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:15.256810 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.256755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:08:15.257884 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:08:15.257860    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0873308_c1eb_4b28_9c25_12a706185f4f.slice/crio-0f0333e7ea17036d6c843b7cde9d1deaaed0c5fd5da28b44e2ea3bc6bc54e2c2 WatchSource:0}: Error finding container 0f0333e7ea17036d6c843b7cde9d1deaaed0c5fd5da28b44e2ea3bc6bc54e2c2: Status 404 returned error can't find the container with id 0f0333e7ea17036d6c843b7cde9d1deaaed0c5fd5da28b44e2ea3bc6bc54e2c2
Jul 07 17:08:15.449715 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.449615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" event={"ID":"a0873308-c1eb-4b28-9c25-12a706185f4f","Type":"ContainerStarted","Data":"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"}
Jul 07 17:08:15.449715 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.449671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" event={"ID":"a0873308-c1eb-4b28-9c25-12a706185f4f","Type":"ContainerStarted","Data":"0f0333e7ea17036d6c843b7cde9d1deaaed0c5fd5da28b44e2ea3bc6bc54e2c2"}
Jul 07 17:08:15.479031 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:15.478977    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" podStartSLOduration=1.4789608140000001 podStartE2EDuration="1.478960814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:15.478544118 +0000 UTC m=+2694.011460761" watchObservedRunningTime="2026-07-07 17:08:15.478960814 +0000 UTC m=+2694.011877457"
Jul 07 17:08:25.108510 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:25.108455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:25.108971 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:25.108555    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:25.124459 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:25.124429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:08:25.511021 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:08:25.510930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:09:21.074509 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:21.074464    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:09:36.096281 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.096244    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" podUID="a0873308-c1eb-4b28-9c25-12a706185f4f" containerName="main" containerID="cri-o://5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3" gracePeriod=45
Jul 07 17:09:36.353659 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.353593    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:09:36.439926 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.439886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440105 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.439953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440105 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.439995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn9fb\" (UniqueName: \"kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440105 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440105 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440105 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache\") pod \"a0873308-c1eb-4b28-9c25-12a706185f4f\" (UID: \"a0873308-c1eb-4b28-9c25-12a706185f4f\") "
Jul 07 17:09:36.440341 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440304    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:36.440341 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440318    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home" (OuterVolumeSpecName: "home") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:36.440455 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440431    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:36.440570 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440553    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-home\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.440633 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440575    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-tmp-dir\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.440633 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.440589    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-model-cache\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.442158 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.442134    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:36.442507 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.442487    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm" (OuterVolumeSpecName: "dshm") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:36.442551 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.442492    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb" (OuterVolumeSpecName: "kube-api-access-sn9fb") pod "a0873308-c1eb-4b28-9c25-12a706185f4f" (UID: "a0873308-c1eb-4b28-9c25-12a706185f4f"). InnerVolumeSpecName "kube-api-access-sn9fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:36.541432 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.541394    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0873308-c1eb-4b28-9c25-12a706185f4f-dshm\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.541432 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.541427    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9fb\" (UniqueName: \"kubernetes.io/projected/a0873308-c1eb-4b28-9c25-12a706185f4f-kube-api-access-sn9fb\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.541432 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.541438    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0873308-c1eb-4b28-9c25-12a706185f4f-tls-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:36.878569 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.878529    2565 generic.go:356] "Generic (PLEG): container finished" podID="a0873308-c1eb-4b28-9c25-12a706185f4f" containerID="5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3" exitCode=0
Jul 07 17:09:36.878806 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.878581    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"
Jul 07 17:09:36.878806 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.878633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" event={"ID":"a0873308-c1eb-4b28-9c25-12a706185f4f","Type":"ContainerDied","Data":"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"}
Jul 07 17:09:36.878806 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.878683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln" event={"ID":"a0873308-c1eb-4b28-9c25-12a706185f4f","Type":"ContainerDied","Data":"0f0333e7ea17036d6c843b7cde9d1deaaed0c5fd5da28b44e2ea3bc6bc54e2c2"}
Jul 07 17:09:36.878806 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.878707    2565 scope.go:117] "RemoveContainer" containerID="5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"
Jul 07 17:09:36.887955 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.887936    2565 scope.go:117] "RemoveContainer" containerID="5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"
Jul 07 17:09:36.888228 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:09:36.888213    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3\": container with ID starting with 5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3 not found: ID does not exist" containerID="5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"
Jul 07 17:09:36.888279 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.888237    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3"} err="failed to get container status \"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3\": rpc error: code = NotFound desc = could not find container \"5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3\": container with ID starting with 5fb1e9ed73eac4252a52b09faa5b0d4a55652a7994496866a84c3e01077af1c3 not found: ID does not exist"
Jul 07 17:09:36.900746 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.900707    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:09:36.904371 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:36.904346    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/rate-test-model-58e7b2-kserve-5b47485bd7-87pln"]
Jul 07 17:09:38.095913 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:38.095881    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0873308-c1eb-4b28-9c25-12a706185f4f" path="/var/lib/kubelet/pods/a0873308-c1eb-4b28-9c25-12a706185f4f/volumes"
Jul 07 17:09:39.906664 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:39.906628    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:09:39.907120 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:39.906943    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy" containerID="cri-o://4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b" gracePeriod=30
Jul 07 17:09:45.163926 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.163894    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:09:45.324081 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324081 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trtrn\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324339 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324637 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data\") pod \"a588edf8-0338-4391-9db3-552502f60506\" (UID: \"a588edf8-0338-4391-9db3-552502f60506\") "
Jul 07 17:09:45.324637 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324399    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.324637 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324536    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.324852 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.324830    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.325029 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.325000    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:45.325029 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.325012    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.325172 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.325014    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.325632 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.325604    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data" (OuterVolumeSpecName: "istio-data") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.326522 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.326491    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 17:09:45.326625 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.326588    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token" (OuterVolumeSpecName: "istio-token") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:45.326846 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.326824    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn" (OuterVolumeSpecName: "kube-api-access-trtrn") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "kube-api-access-trtrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:45.326846 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.326827    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a588edf8-0338-4391-9db3-552502f60506" (UID: "a588edf8-0338-4391-9db3-552502f60506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.426419 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426370    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trtrn\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-kube-api-access-trtrn\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426419 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426414    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426419 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426425    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426419 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426433    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588edf8-0338-4391-9db3-552502f60506-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426694 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426440    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588edf8-0338-4391-9db3-552502f60506-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426694 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426448    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588edf8-0338-4391-9db3-552502f60506-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.426694 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.426455    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588edf8-0338-4391-9db3-552502f60506-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.946735 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.946649    2565 generic.go:356] "Generic (PLEG): container finished" podID="a588edf8-0338-4391-9db3-552502f60506" containerID="4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b" exitCode=0
Jul 07 17:09:45.947153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.946737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerDied","Data":"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"}
Jul 07 17:09:45.947153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.946820    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"
Jul 07 17:09:45.947153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.946886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78" event={"ID":"a588edf8-0338-4391-9db3-552502f60506","Type":"ContainerDied","Data":"7ba597ac5d30559cb691bff60100bc9715f591be5d86ae44ba6584891e12e46f"}
Jul 07 17:09:45.947153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.946917    2565 scope.go:117] "RemoveContainer" containerID="4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"
Jul 07 17:09:45.960878 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.960854    2565 scope.go:117] "RemoveContainer" containerID="f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"
Jul 07 17:09:45.973245 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.973210    2565 scope.go:117] "RemoveContainer" containerID="4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"
Jul 07 17:09:45.973774 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:09:45.973745    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b\": container with ID starting with 4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b not found: ID does not exist" containerID="4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"
Jul 07 17:09:45.974006 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.973806    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b"} err="failed to get container status \"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b\": rpc error: code = NotFound desc = could not find container \"4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b\": container with ID starting with 4107b7bf011f0f39901f60f0e131571220061cb41d23efae3fa64755ef34332b not found: ID does not exist"
Jul 07 17:09:45.974006 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.973834    2565 scope.go:117] "RemoveContainer" containerID="f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"
Jul 07 17:09:45.974241 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:09:45.974223    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35\": container with ID starting with f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35 not found: ID does not exist" containerID="f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"
Jul 07 17:09:45.974289 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.974259    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35"} err="failed to get container status \"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35\": rpc error: code = NotFound desc = could not find container \"f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35\": container with ID starting with f3a450ee5e350684235421ed06d337e480e9223ab95af1d99a4c84919efe5c35 not found: ID does not exist"
Jul 07 17:09:45.975212 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.975181    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:09:45.979584 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:45.979507    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-58e7b2-openshift-default-5c885b9979-9bw78"]
Jul 07 17:09:46.095415 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:46.095373    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a588edf8-0338-4391-9db3-552502f60506" path="/var/lib/kubelet/pods/a588edf8-0338-4391-9db3-552502f60506/volumes"
Jul 07 17:09:47.910563 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:47.910527    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:09:47.911050 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:47.910849    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy" containerID="cri-o://0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d" gracePeriod=30
Jul 07 17:09:53.155638 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.155613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:09:53.309078 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.308989    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309078 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309078 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309062    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkldq\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309368 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309296    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs\") pod \"863e23a6-d873-478e-bae2-806125b70ec1\" (UID: \"863e23a6-d873-478e-bae2-806125b70ec1\") "
Jul 07 17:09:53.309655 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309381    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:53.309655 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309589    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:53.309752 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309659    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.309752 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309722    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:53.309878 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.309826    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:53.310277 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.310247    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data" (OuterVolumeSpecName: "istio-data") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:53.311379 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.311346    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:53.311618 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.311581    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token" (OuterVolumeSpecName: "istio-token") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:53.311698 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.311677    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq" (OuterVolumeSpecName: "kube-api-access-bkldq") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "kube-api-access-bkldq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:53.311741 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.311711    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "863e23a6-d873-478e-bae2-806125b70ec1" (UID: "863e23a6-d873-478e-bae2-806125b70ec1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 17:09:53.410439 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410399    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-istio-token\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410439 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410432    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-envoy\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410447    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkldq\" (UniqueName: \"kubernetes.io/projected/863e23a6-d873-478e-bae2-806125b70ec1-kube-api-access-bkldq\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410460    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863e23a6-d873-478e-bae2-806125b70ec1-istiod-ca-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410471    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-istio-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410482    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-credential-socket\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410492    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863e23a6-d873-478e-bae2-806125b70ec1-istio-podinfo\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.410671 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.410502    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863e23a6-d873-478e-bae2-806125b70ec1-workload-certs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\""
Jul 07 17:09:53.507129 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.507094    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cdwlb_841928fa-7995-4140-b430-ddc279ffe2ad/manager/0.log"
Jul 07 17:09:53.641366 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.641335    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n8mhx_0f0e39de-e6f3-4322-9a6a-ed33b56f85f7/manager/0.log"
Jul 07 17:09:53.784162 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.784126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-667b548c96-c47mq_f29c39ec-0805-4e71-8cab-d6fc364c3ac9/maas-api/0.log"
Jul 07 17:09:53.989161 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.989071    2565 generic.go:356] "Generic (PLEG): container finished" podID="863e23a6-d873-478e-bae2-806125b70ec1" containerID="0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d" exitCode=0
Jul 07 17:09:53.989161 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.989127    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"
Jul 07 17:09:53.989396 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.989154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerDied","Data":"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"}
Jul 07 17:09:53.989396 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.989217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d" event={"ID":"863e23a6-d873-478e-bae2-806125b70ec1","Type":"ContainerDied","Data":"5a3137447e2dca6bbfda5596e8a1356762a6d9e273b17d34d43e4e3b166f6172"}
Jul 07 17:09:53.989396 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.989247    2565 scope.go:117] "RemoveContainer" containerID="0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"
Jul 07 17:09:53.998930 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:53.998914    2565 scope.go:117] "RemoveContainer" containerID="de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"
Jul 07 17:09:54.006853 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.006835    2565 scope.go:117] "RemoveContainer" containerID="0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"
Jul 07 17:09:54.007136 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:09:54.007117    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d\": container with ID starting with 0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d not found: ID does not exist" containerID="0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"
Jul 07 17:09:54.007204 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.007143    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d"} err="failed to get container status \"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d\": rpc error: code = NotFound desc = could not find container \"0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d\": container with ID starting with 0c922ad0193f25aeb6db5fbd42e5bd4d1229aa3398034b5a2adb194676c5e28d not found: ID does not exist"
Jul 07 17:09:54.007204 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.007162    2565 scope.go:117] "RemoveContainer" containerID="de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"
Jul 07 17:09:54.007412 ip-10-0-140-28 kubenswrapper[2565]: E0707 17:09:54.007390    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c\": container with ID starting with de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c not found: ID does not exist" containerID="de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"
Jul 07 17:09:54.007470 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.007423    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c"} err="failed to get container status \"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c\": rpc error: code = NotFound desc = could not find container \"de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c\": container with ID starting with de41cbe5405d872daf2546b43b01023f1693ab108336d8454ff1815b88a2f22c not found: ID does not exist"
Jul 07 17:09:54.023115 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.023088    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:09:54.038716 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.038686    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-fa4616-openshift-default-644bc5945f-vhw7d"]
Jul 07 17:09:54.095721 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.095693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863e23a6-d873-478e-bae2-806125b70ec1" path="/var/lib/kubelet/pods/863e23a6-d873-478e-bae2-806125b70ec1/volumes"
Jul 07 17:09:54.316237 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.316212    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8jbcp_e14eda58-c8d5-47b8-9622-f57afb62be5e/server/0.log"
Jul 07 17:09:54.443275 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.443250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-46hj7_61ccf201-2c95-45b5-a08e-7776ad2134e8/manager/0.log"
Jul 07 17:09:54.732007 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.731923    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-6pz6b_44532bf2-29ef-42b8-8a92-a5a043ba625f/manager/0.log"
Jul 07 17:09:54.739564 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:54.739546    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-6pz6b_44532bf2-29ef-42b8-8a92-a5a043ba625f/copy-manifests/0.log"
Jul 07 17:09:56.520263 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:56.520237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bs26p_2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17/manager/0.log"
Jul 07 17:09:56.632043 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:56.632010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tp25f_dee4984f-5355-4343-a1e3-8c670bc29588/manager/0.log"
Jul 07 17:09:56.871017 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:56.870975    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pxfjk_bb5426e9-c3ed-4431-b338-9ccfcb77a04d/registry-server/0.log"
Jul 07 17:09:57.003271 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:57.003229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lckwd_22421eb8-6052-41ca-b805-57d81ff82bb8/manager/0.log"
Jul 07 17:09:57.258174 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:57.258085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-phvsn_241ee8c1-0e6e-4cc3-aeca-634a1ebacab6/manager/0.log"
Jul 07 17:09:57.656995 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:57.656963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxgxb_ef8bf763-6f0b-42b6-9f92-6f9a4c599851/istio-proxy/0.log"
Jul 07 17:09:57.915008 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:57.914921    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-5t7cd_063ae709-1e25-408e-b2ac-1c21f34fefb1/kube-auth-proxy/0.log"
Jul 07 17:09:58.175349 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:58.175281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/1.log"
Jul 07 17:09:59.093053 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:59.093024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-knk4g_ab86ff7e-6a1c-48a9-b7ee-c1a5e6f24068/main/0.log"
Jul 07 17:09:59.357413 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:59.357346    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9vbkc_8c4b8acb-f560-4b87-bac3-41c8baf74f7f/main/0.log"
Jul 07 17:09:59.623304 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:09:59.623228    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tcpxf_9a04425b-b330-4c69-9e1f-f61657c72ed8/main/0.log"
Jul 07 17:10:06.910719 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:06.910689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6rrzf_5ee74c68-b023-4f56-8591-8449f020992f/global-pull-secret-syncer/0.log"
Jul 07 17:10:07.113153 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:07.113122    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tw2zg_18b23933-59e6-4828-9915-724175f870ca/konnectivity-agent/0.log"
Jul 07 17:10:07.211879 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:07.211765    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-28.ec2.internal_6eb5e9c0a661aaef32578bdf54d1da55/haproxy/0.log"
Jul 07 17:10:12.392829 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:12.392780    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bs26p_2f31b8a0-1741-4fa6-b7c1-7a80d63b5a17/manager/0.log"
Jul 07 17:10:12.422161 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:12.422137    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tp25f_dee4984f-5355-4343-a1e3-8c670bc29588/manager/0.log"
Jul 07 17:10:12.509321 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:12.509296    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pxfjk_bb5426e9-c3ed-4431-b338-9ccfcb77a04d/registry-server/0.log"
Jul 07 17:10:12.712216 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:12.712139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lckwd_22421eb8-6052-41ca-b805-57d81ff82bb8/manager/0.log"
Jul 07 17:10:12.872327 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:12.872297    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-phvsn_241ee8c1-0e6e-4cc3-aeca-634a1ebacab6/manager/0.log"
Jul 07 17:10:14.834257 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:14.834221    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d4cd94785-5gj62_672ae99d-6d82-4b98-8af7-0f8d842d6bf3/metrics-server/0.log"
Jul 07 17:10:15.106218 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.106152    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5jkq_1cad0191-16ea-47bd-91a7-cdfe66919fb2/node-exporter/0.log"
Jul 07 17:10:15.127846 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.127810    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5jkq_1cad0191-16ea-47bd-91a7-cdfe66919fb2/kube-rbac-proxy/0.log"
Jul 07 17:10:15.150347 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.150327    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5jkq_1cad0191-16ea-47bd-91a7-cdfe66919fb2/init-textfile/0.log"
Jul 07 17:10:15.709118 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.709078    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/thanos-query/0.log"
Jul 07 17:10:15.735383 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.735361    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/kube-rbac-proxy-web/0.log"
Jul 07 17:10:15.761769 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.761735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/kube-rbac-proxy/0.log"
Jul 07 17:10:15.788377 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.788356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/prom-label-proxy/0.log"
Jul 07 17:10:15.815309 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.815285    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/kube-rbac-proxy-rules/0.log"
Jul 07 17:10:15.841607 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:15.841587    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c6977688-rbtjb_36593203-87af-48c7-a09b-b68513d1ab10/kube-rbac-proxy-metrics/0.log"
Jul 07 17:10:16.649960 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.649927    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"]
Jul 07 17:10:16.650378 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0873308-c1eb-4b28-9c25-12a706185f4f" containerName="main"
Jul 07 17:10:16.650461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650404    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0873308-c1eb-4b28-9c25-12a706185f4f" containerName="main"
Jul 07 17:10:16.650461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650434    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650464    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650569    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0873308-c1eb-4b28-9c25-12a706185f4f" containerName="main"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650581    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650601    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="863e23a6-d873-478e-bae2-806125b70ec1" containerName="istio-proxy"
Jul 07 17:10:16.650701 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.650612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a588edf8-0338-4391-9db3-552502f60506" containerName="istio-proxy"
Jul 07 17:10:16.653840 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.653818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.656325 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.656307    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkp5f\"/\"kube-root-ca.crt\""
Jul 07 17:10:16.656897 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.656875    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkp5f\"/\"openshift-service-ca.crt\""
Jul 07 17:10:16.658485 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.658464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nkp5f\"/\"default-dockercfg-4vg9d\""
Jul 07 17:10:16.667204 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.667185    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"]
Jul 07 17:10:16.698117 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.698094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlqs\" (UniqueName: \"kubernetes.io/projected/ee139b4f-4820-4e96-9fa4-a443a8377d51-kube-api-access-5xlqs\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.698267 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.698140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-sys\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.698267 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.698164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-lib-modules\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.698267 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.698219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-proc\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.698267 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.698264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-podres\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799247 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlqs\" (UniqueName: \"kubernetes.io/projected/ee139b4f-4820-4e96-9fa4-a443a8377d51-kube-api-access-5xlqs\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799365 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-sys\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799365 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-lib-modules\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799365 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-proc\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799365 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-podres\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799546 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799404    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-sys\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799546 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799439    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-lib-modules\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799546 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799448    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-podres\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.799546 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.799439    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee139b4f-4820-4e96-9fa4-a443a8377d51-proc\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.808572 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.808547    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlqs\" (UniqueName: \"kubernetes.io/projected/ee139b4f-4820-4e96-9fa4-a443a8377d51-kube-api-access-5xlqs\") pod \"perf-node-gather-daemonset-8mgpr\" (UID: \"ee139b4f-4820-4e96-9fa4-a443a8377d51\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:16.964652 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:16.964591    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:17.101716 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:17.101686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"]
Jul 07 17:10:17.102965 ip-10-0-140-28 kubenswrapper[2565]: W0707 17:10:17.102939    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee139b4f_4820_4e96_9fa4_a443a8377d51.slice/crio-77d98aae73531063300eeaed463b40752b4cecb9d62ec67353787bc2f78dd7e9 WatchSource:0}: Error finding container 77d98aae73531063300eeaed463b40752b4cecb9d62ec67353787bc2f78dd7e9: Status 404 returned error can't find the container with id 77d98aae73531063300eeaed463b40752b4cecb9d62ec67353787bc2f78dd7e9
Jul 07 17:10:17.112356 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:17.112332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr" event={"ID":"ee139b4f-4820-4e96-9fa4-a443a8377d51","Type":"ContainerStarted","Data":"77d98aae73531063300eeaed463b40752b4cecb9d62ec67353787bc2f78dd7e9"}
Jul 07 17:10:18.118304 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:18.118274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr" event={"ID":"ee139b4f-4820-4e96-9fa4-a443a8377d51","Type":"ContainerStarted","Data":"455ae43b5e6e30d115f54222407e77fcb772e99b58b9ec9570b5664e6bb38d52"}
Jul 07 17:10:18.118672 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:18.118327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:18.138578 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:18.138533    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr" podStartSLOduration=2.1385193 podStartE2EDuration="2.1385193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:10:18.136613274 +0000 UTC m=+2816.669529929" watchObservedRunningTime="2026-07-07 17:10:18.1385193 +0000 UTC m=+2816.671435971"
Jul 07 17:10:19.963778 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:19.963746    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnwzn_05439776-992c-487e-b154-0cb34885da99/dns/0.log"
Jul 07 17:10:19.993721 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:19.993689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnwzn_05439776-992c-487e-b154-0cb34885da99/kube-rbac-proxy/0.log"
Jul 07 17:10:20.085232 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:20.085209    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fn2kg_249ad43d-fc4a-4a80-972b-e88b626b20fa/dns-node-resolver/0.log"
Jul 07 17:10:20.744021 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:20.743991    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c62q6_46480d98-8a3a-4a69-8214-db6d53f2f9d3/node-ca/0.log"
Jul 07 17:10:21.840672 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:21.840642    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxgxb_ef8bf763-6f0b-42b6-9f92-6f9a4c599851/istio-proxy/0.log"
Jul 07 17:10:22.070094 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:22.070065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-5t7cd_063ae709-1e25-408e-b2ac-1c21f34fefb1/kube-auth-proxy/0.log"
Jul 07 17:10:22.129174 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:22.129088    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/0.log"
Jul 07 17:10:22.184777 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:22.184750    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/1.log"
Jul 07 17:10:23.044721 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:23.044693    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sp2sb_f09972be-b56f-4b94-9583-6efcaeb8ce09/serve-healthcheck-canary/0.log"
Jul 07 17:10:23.577736 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:23.577707    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sc9g_f22ab0f8-0c44-4106-afa3-13619b6b0dc9/kube-rbac-proxy/0.log"
Jul 07 17:10:23.605303 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:23.605279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sc9g_f22ab0f8-0c44-4106-afa3-13619b6b0dc9/exporter/0.log"
Jul 07 17:10:23.629998 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:23.629978    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sc9g_f22ab0f8-0c44-4106-afa3-13619b6b0dc9/extractor/0.log"
Jul 07 17:10:24.132858 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:24.132836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-8mgpr"
Jul 07 17:10:26.208749 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:26.208722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cdwlb_841928fa-7995-4140-b430-ddc279ffe2ad/manager/0.log"
Jul 07 17:10:26.467539 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:26.467515    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n8mhx_0f0e39de-e6f3-4322-9a6a-ed33b56f85f7/manager/0.log"
Jul 07 17:10:26.539186 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:26.539164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-667b548c96-c47mq_f29c39ec-0805-4e71-8cab-d6fc364c3ac9/maas-api/0.log"
Jul 07 17:10:26.822461 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:26.822441    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8jbcp_e14eda58-c8d5-47b8-9622-f57afb62be5e/server/0.log"
Jul 07 17:10:26.943711 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:26.943689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-46hj7_61ccf201-2c95-45b5-a08e-7776ad2134e8/manager/0.log"
Jul 07 17:10:27.134556 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.134533    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxgxb_ef8bf763-6f0b-42b6-9f92-6f9a4c599851/istio-proxy/0.log"
Jul 07 17:10:27.172615 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.172598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-6pz6b_44532bf2-29ef-42b8-8a92-a5a043ba625f/manager/0.log"
Jul 07 17:10:27.219082 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.219032    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-6pz6b_44532bf2-29ef-42b8-8a92-a5a043ba625f/copy-manifests/0.log"
Jul 07 17:10:27.348852 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.348829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-5t7cd_063ae709-1e25-408e-b2ac-1c21f34fefb1/kube-auth-proxy/0.log"
Jul 07 17:10:27.384642 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.384619    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/0.log"
Jul 07 17:10:27.439707 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:27.439678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/1.log"
Jul 07 17:10:28.719305 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:28.719282    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxgxb_ef8bf763-6f0b-42b6-9f92-6f9a4c599851/istio-proxy/0.log"
Jul 07 17:10:28.892865 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:28.892792    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5nmtw_453f65f2-e43f-403f-b2ea-257c21e35e1c/openshift-lws-operator/0.log"
Jul 07 17:10:28.940320 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:28.940293    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-5t7cd_063ae709-1e25-408e-b2ac-1c21f34fefb1/kube-auth-proxy/0.log"
Jul 07 17:10:28.982999 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:28.982979    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/0.log"
Jul 07 17:10:29.039609 ip-10-0-140-28 kubenswrapper[2565]: I0707 17:10:29.039586    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wh6l4_99f63330-fb5b-4500-813e-978da828aa64/istio-proxy/1.log"