Jul 10 02:37:59.179406 ip-10-0-136-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 02:37:59.179424 ip-10-0-136-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 02:37:59.179434 ip-10-0-136-90 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 02:37:59.179788 ip-10-0-136-90 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 02:38:09.278250 ip-10-0-136-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 02:38:09.278267 ip-10-0-136-90 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d38cd10ec4064f8db0f76c9e999c4e9c --
Jul 10 02:40:06.071441 ip-10-0-136-90 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 02:40:06.422388 ip-10-0-136-90 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 02:40:06.422388 ip-10-0-136-90 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 02:40:06.422388 ip-10-0-136-90 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 02:40:06.422388 ip-10-0-136-90 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 02:40:06.422388 ip-10-0-136-90 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 02:40:06.424870 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.424699    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 02:40:06.430224 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430201    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.430224 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430220    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.430224 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430225    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.430224 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430231    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430235    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430242    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430246    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430251    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430255    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430259    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430262    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430267    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430271    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430275    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430279    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430283    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430288    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430292    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430297    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430301    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430305    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430310    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430315    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.430489 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430322    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430328    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430334    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430339    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430343    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430348    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430352    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430356    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430361    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430366    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430370    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430375    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430380    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430384    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430390    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430395    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430400    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430405    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430410    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430416    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.431390 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430421    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430428    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430432    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430437    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430441    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430446    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430450    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430455    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430461    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430466    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430471    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430475    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430480    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430485    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430489    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430493    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430498    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430503    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430508    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430512    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.432298 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430516    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430521    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430525    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430531    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430535    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430539    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430550    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430556    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430562    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430568    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430573    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430578    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430585    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430590    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430595    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430600    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430606    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430610    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430615    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430619    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.433155 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430624    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430628    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430633    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430637    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430642    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430647    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430651    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430656    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430661    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430666    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430673    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430679    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430684    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430688    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430693    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.430697    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.431976    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.431986    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.431992    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.434106 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.431997    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432002    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432007    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432012    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432017    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432022    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432027    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432032    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432037    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432042    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432046    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432051    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432056    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432060    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432065    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432070    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432074    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432079    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432083    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432088    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.434974 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432093    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432098    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432102    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432107    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432112    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432116    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432121    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432125    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432130    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432134    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432138    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432143    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432148    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432153    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432158    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432162    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432167    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432199    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432204    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432216    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.435751 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432222    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432227    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432232    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432237    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432241    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432245    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432252    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432259    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432264    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432269    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432274    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432278    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432283    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432288    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432292    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432296    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432301    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432305    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432311    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.436367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432318    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432324    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432330    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432335    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432341    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432346    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432350    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432355    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432359    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432364    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432369    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432373    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432377    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432383    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432389    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432393    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432399    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432403    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432408    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.436970 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432416    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432422    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432428    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432432    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432438    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432443    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432448    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432453    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432458    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432462    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432466    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432471    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432475    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432480    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432485    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432489    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432494    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.432499    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433200    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433210    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.437687 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433216    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433220    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433225    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433233    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433239    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433244    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433251    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433256    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433262    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433267    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433272    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433277    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433289    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433294    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433298    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433303    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433308    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433312    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433316    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.438251 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433321    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433325    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433332    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433338    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433344    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433349    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433354    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433358    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433363    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433367    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433371    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433376    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433380    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433385    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433389    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433393    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433398    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433402    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433406    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433411    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.438862 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433422    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433429    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433437    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433441    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433446    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433450    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433455    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433459    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433463    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433468    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433472    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433477    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433481    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433485    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433490    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433494    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433498    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433503    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433508    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433512    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.439467 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433517    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433521    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433526    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433531    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433535    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433539    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433544    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433548    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433553    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433558    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433562    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433567    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433571    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433577    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433581    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433588    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433593    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433597    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433602    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433606    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.439977 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433611    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433615    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433620    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433624    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433628    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433633    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433637    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433642    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433647    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433652    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433656    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433661    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433665    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433670    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433674    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433678    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433683    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.433687    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433804    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433817    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433829    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 02:40:06.440540 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433836    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433844    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433850    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433858    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433865    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433872    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433877    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433884    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433889    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433895    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433901    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433906    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433912    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433917    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433922    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433929    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433933    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433939    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433945    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433952    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433958    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433963    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433969    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 02:40:06.441118 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433974    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433980    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433985    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433991    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.433996    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434001    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434008    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434013    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434018    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434023    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434029    2551 flags.go:64] FLAG: --enable-server="true"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434035    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434043    2551 flags.go:64] FLAG: --event-burst="100"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434048    2551 flags.go:64] FLAG: --event-qps="50"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434053    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434059    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434064    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434071    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434076    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434082    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434087    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434092    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434097    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434102    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434107    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 02:40:06.441814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434112    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434116    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434121    2551 flags.go:64] FLAG: --feature-gates=""
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434138    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434143    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434149    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434155    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434160    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434166    2551 flags.go:64] FLAG: --help="false"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434188    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434194    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434199    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434205    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434212    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434218    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434223    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434229    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434234    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434239    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434244    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434250    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434255    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434261    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434267    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 02:40:06.442635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434272    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434277    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434282    2551 flags.go:64] FLAG: --lock-file=""
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434287    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434292    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434298    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434307    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434313    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434318    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434323    2551 flags.go:64] FLAG: --logging-format="text"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434328    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434336    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434342    2551 flags.go:64] FLAG: --manifest-url=""
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434347    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434355    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434361    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434368    2551 flags.go:64] FLAG: --max-pods="110"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434373    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434379    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434384    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434389    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434394    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434399    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434405    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434420    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 02:40:06.443289 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434426    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434431    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434437    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434442    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434452    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434457    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434462    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434469    2551 flags.go:64] FLAG: --port="10250"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434475    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434480    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0680d00dce676e335"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434486    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434491    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434496    2551 flags.go:64] FLAG: --register-node="true"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434502    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434508    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434514    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434520    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434525    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434531    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434542    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434547    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434552    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434558    2551 flags.go:64] FLAG: --runonce="false"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434564    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434570    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 02:40:06.443941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434575    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434580    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434585    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434591    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434596    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434602    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434609    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434614    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434619    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434624    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434630    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434635    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434640    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434650    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434655    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434661    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434668    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434673    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434678    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434683    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434688    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434695    2551 flags.go:64] FLAG: --v="2"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434702    2551 flags.go:64] FLAG: --version="false"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434709    2551 flags.go:64] FLAG: --vmodule=""
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434715    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 02:40:06.444671 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.434721    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434953    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434960    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434966    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434971    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434976    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434981    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434986    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434991    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.434996    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435001    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435006    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435011    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435018    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435023    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435029    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435034    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435039    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435043    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435048    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.445377 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435052    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435057    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435061    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435067    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435072    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435076    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435081    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435085    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435090    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435094    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435099    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435104    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435108    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435115    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435120    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435124    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435129    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435133    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435138    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.445890 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435145    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435151    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435157    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435163    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435167    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435193    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435200    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435205    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435211    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435215    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435220    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435225    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435229    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435234    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435239    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435243    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435248    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435253    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435258    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435263    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.446415 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435267    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435272    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435276    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435281    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435285    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435290    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435296    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435300    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435305    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435310    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435314    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435319    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435323    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435328    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435333    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435338    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435342    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435346    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435353    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435357    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.446927 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435362    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435367    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435371    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435376    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435380    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435385    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435391    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435398    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435403    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435408    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435413    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435417    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435422    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435426    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435431    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435435    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435440    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435444    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435451    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.447470 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435458    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.435464    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.436163    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.443771    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.443795    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.443818    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.443828    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443915    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443921    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443924    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443927    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443930    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443933    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443936    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.447967 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443939    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443942    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443946    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443949    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443952    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443955    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443958    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443961    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443964    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443967    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443970    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443973    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443976    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443979    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443981    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443984    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443987    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443990    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443992    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443995    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.448367 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.443998    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444001    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444011    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444014    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444017    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444019    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444022    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444026    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444028    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444033    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444037    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444041    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444044    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444047    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444050    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444053    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444056    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444059    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444062    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.448874 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444066    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444068    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444072    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444075    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444078    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444080    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444083    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444086    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444089    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444091    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444094    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444097    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444099    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444102    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444105    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444108    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444117    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444120    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444124    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444126    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.449387 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444129    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444132    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444135    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444137    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444140    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444143    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444146    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444149    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444152    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444154    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444157    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444160    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444162    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444165    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444168    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444185    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444188    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444193    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444197    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.449916 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444200    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444203    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444205    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444208    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444211    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444214    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444217    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444220    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444223    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444227    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444237    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444241    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444244    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444247    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.450433 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.444253    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444431    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444438    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444443    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444447    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444452    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444456    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444459    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444463    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444466    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444469    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444472    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444475    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444477    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444480    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444483    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444486    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444488    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444491    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 02:40:06.450818 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444494    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444497    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444500    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444503    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444506    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444509    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444512    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444514    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444517    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444520    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444531    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444535    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444538    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444542    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444545    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444548    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444551    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444554    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444564    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 02:40:06.451335 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444567    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444570    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444573    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444575    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444578    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444580    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444583    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444586    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444588    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444591    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444594    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444597    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444599    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444602    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444605    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444608    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444610    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444613    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444616    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 02:40:06.451822 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444618    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444621    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444624    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444627    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444629    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444638    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444641    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444644    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444647    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444650    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444655    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444658    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444661    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444664    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444666    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444669    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444671    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444674    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444677    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444680    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 02:40:06.452334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444683    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444686    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444689    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444691    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444694    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444697    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444699    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444702    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444705    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444708    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444710    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444713    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444716    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444719    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444721    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444724    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444727    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444729    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444738    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444741    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 02:40:06.452858 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444744    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444746    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:06.444750    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.444755    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.444910    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.447012    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.448397    2551 server.go:1019] "Starting client certificate rotation"
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.448504    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.448542    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 02:40:06.453402 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.450651    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 02:40:06.453652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.453443    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 02:40:06.461169 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.461147    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 10 02:40:06.463271 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.463251    2551 log.go:25] "Validated CRI v1 image API"
Jul 10 02:40:06.464163 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.464137    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 02:40:06.468816 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.468793    2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e444f7f-54d6-47fe-af5c-7ca5add26e30:/dev/nvme0n1p4 fead43e9-bcdc-49ab-8f70-3a0cb4898f68:/dev/nvme0n1p3]
Jul 10 02:40:06.468890 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.468815    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 02:40:06.475083 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.474951    2551 manager.go:216] Machine: {Timestamp:2026-07-10 02:40:06.472898816 +0000 UTC m=+0.285976902 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100253 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e426fd64878d4c62c1a4647460912 SystemUUID:ec2e426f-d648-78d4-c62c-1a4647460912 BootID:d38cd10e-c406-4f8d-b0f7-6c9e999c4e9c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:73:e6:db:7a:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:73:e6:db:7a:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:ff:64:f2:15:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 02:40:06.475083 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.475074    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 02:40:06.475246 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.475186    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 02:40:06.477971 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.477932    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 02:40:06.478142 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.477973    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 02:40:06.479102 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.479090    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 02:40:06.479142 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.479105    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 02:40:06.479142 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.479114    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 02:40:06.479713 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.479701    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 02:40:06.480425 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.480412    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 02:40:06.482919 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.482899    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 02:40:06.483053 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483043    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 02:40:06.483094 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483075    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 02:40:06.483124 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483100    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 02:40:06.483157 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483123    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 02:40:06.483157 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483136    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 10 02:40:06.483157 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483147    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 02:40:06.483310 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483287    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 02:40:06.483686 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483670    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 02:40:06.483809 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.483692    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 02:40:06.485923 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.485894    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 02:40:06.487240 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487224    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 02:40:06.487300 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487259    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 02:40:06.487832 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487820    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 02:40:06.487867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487838    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 02:40:06.487867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487846    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 02:40:06.487867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487852    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 02:40:06.487867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487861    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 02:40:06.487867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.487866    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 02:40:06.488081 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488072    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 02:40:06.488081 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488081    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 02:40:06.488138 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488088    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 02:40:06.488138 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488094    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 02:40:06.488138 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488103    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 02:40:06.488138 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488112    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 02:40:06.488718 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488706    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 02:40:06.488718 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.488718    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 02:40:06.492524 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.492509    2551 server.go:1268] "Started kubelet"
Jul 10 02:40:06.492637 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.492599    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 02:40:06.492747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.492655    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 02:40:06.492976 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.492621    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 02:40:06.493533 ip-10-0-136-90 systemd[1]: Started Kubernetes Kubelet.
Jul 10 02:40:06.494301 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.494121    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 02:40:06.497844 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.497820    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 02:40:06.497844 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.497824    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 02:40:06.498010 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.497914    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 02:40:06.498396 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.498372    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 02:40:06.499051 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.499034    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 02:40:06.500420 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.500394    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 02:40:06.500420 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.500410    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 02:40:06.500568 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.500502    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.501056 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501036    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 02:40:06.501056 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501036    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 02:40:06.501266 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501063    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 02:40:06.501266 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501224    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 02:40:06.501266 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501235    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 02:40:06.501266 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.500137    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-90.ec2.internal.18c0cd52762e6e87  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-90.ec2.internal,UID:ip-10-0-136-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-90.ec2.internal,},FirstTimestamp:2026-07-10 02:40:06.492483207 +0000 UTC m=+0.305561292,LastTimestamp:2026-07-10 02:40:06.492483207 +0000 UTC m=+0.305561292,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-90.ec2.internal,}"
Jul 10 02:40:06.501549 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501531    2551 factory.go:153] Registering CRI-O factory
Jul 10 02:40:06.501618 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501553    2551 factory.go:223] Registration of the crio container factory successfully
Jul 10 02:40:06.501618 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501613    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 02:40:06.501709 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501627    2551 factory.go:55] Registering systemd factory
Jul 10 02:40:06.501709 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501637    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 10 02:40:06.501709 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501683    2551 factory.go:103] Registering Raw factory
Jul 10 02:40:06.501842 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.501711    2551 manager.go:1196] Started watching for new ooms in manager
Jul 10 02:40:06.502484 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.502465    2551 manager.go:349] Starting recovery of all containers
Jul 10 02:40:06.504254 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.504205    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 02:40:06.504354 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.504284    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 02:40:06.512270 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.512059    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcsxv"
Jul 10 02:40:06.514679 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.514652    2551 manager.go:354] Recovery completed
Jul 10 02:40:06.519651 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.519636    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.520044 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.520023    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcsxv"
Jul 10 02:40:06.521759 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.521743    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.521805 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.521777    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.521805 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.521791    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.522454 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.522439    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 02:40:06.522530 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.522456    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 02:40:06.522530 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.522475    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 02:40:06.522530 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.522440    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.523365 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523348    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.523441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523377    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.523441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523387    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.523604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523589    2551 policy_none.go:49] "None policy: Start"
Jul 10 02:40:06.523638 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523611    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 02:40:06.523638 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.523628    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 02:40:06.524353 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.524267    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-90.ec2.internal.18c0cd5277ed2d5d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-90.ec2.internal,UID:ip-10-0-136-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-90.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-90.ec2.internal,},FirstTimestamp:2026-07-10 02:40:06.521761117 +0000 UTC m=+0.334839202,LastTimestamp:2026-07-10 02:40:06.521761117 +0000 UTC m=+0.334839202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-90.ec2.internal,}"
Jul 10 02:40:06.524549 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.524537    2551 policy_none.go:47] "Start"
Jul 10 02:40:06.559951 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.559927    2551 manager.go:337] "Starting Device Plugin manager"
Jul 10 02:40:06.560091 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.559980    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 02:40:06.560091 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.559995    2551 server.go:87] "Starting device plugin registration server"
Jul 10 02:40:06.560355 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.560341    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 02:40:06.560392 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.560357    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 02:40:06.560590 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.560481    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 02:40:06.560590 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.560589    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 02:40:06.560742 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.560599    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 02:40:06.561195 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.561171    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 02:40:06.561293 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.561233    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.660881 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.660788    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.661907 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.661837    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.661907 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.661875    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.661907 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.661887    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.662084 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.661918    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.670770 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.670745    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.670902 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.670778    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-90.ec2.internal\": node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.679750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.679723    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 02:40:06.681119 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.681093    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 02:40:06.681255 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.681132    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 02:40:06.681255 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.681156    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 02:40:06.681255 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.681216    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 02:40:06.683807 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.683772    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:06.686213 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.686194    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.781645 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.781595    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"]
Jul 10 02:40:06.781823 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.781712    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.782820 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.782797    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.782938 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.782836    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.782938 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.782848    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.784176 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.784161    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.784306 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.784287    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.784356 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.784324    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.786271 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.786253    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.786453 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.786439    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.786504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.786468    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.786504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.786481    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.787477 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.787459    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.787569 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.787485    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.787569 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.787496    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.788548 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.788530    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.788633 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.788563    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 02:40:06.789316 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.789299    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 02:40:06.789398 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.789327    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 02:40:06.789398 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.789338    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeHasSufficientPID"
Jul 10 02:40:06.803566 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.803535    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.803704 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.803570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.803704 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.803596    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a70112095ce53725b3f8952b09deb60-config\") pod \"kube-apiserver-proxy-ip-10-0-136-90.ec2.internal\" (UID: \"0a70112095ce53725b3f8952b09deb60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.815306 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.815279    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-90.ec2.internal\" not found" node="ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.820294 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.820276    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-90.ec2.internal\" not found" node="ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.887242 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.887209    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:06.904358 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.904325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.904469 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.904446    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.905022 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.905002    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.905064 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.905039    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a70112095ce53725b3f8952b09deb60-config\") pod \"kube-apiserver-proxy-ip-10-0-136-90.ec2.internal\" (UID: \"0a70112095ce53725b3f8952b09deb60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.905101 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.905085    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a70112095ce53725b3f8952b09deb60-config\") pod \"kube-apiserver-proxy-ip-10-0-136-90.ec2.internal\" (UID: \"0a70112095ce53725b3f8952b09deb60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.905135 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:06.905094    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331356f040de96b4d1948c4d6e1935b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal\" (UID: \"331356f040de96b4d1948c4d6e1935b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:06.988482 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:06.988413    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.094697 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.094650    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.117846 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.117799    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:07.122845 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.122819    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:07.196843 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.196802    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.297090 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.297029    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.316086 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.316061    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:07.398607 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.398563    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.447994 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.447964    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 02:40:07.448580 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.448136    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 02:40:07.448580 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.448151    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 02:40:07.499690 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.499656    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.524348 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.524303    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 02:35:07 +0000 UTC" deadline="2028-02-29 18:37:48.045748253 +0000 UTC"
Jul 10 02:40:07.524348 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.524337    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14391h57m40.52141429s"
Jul 10 02:40:07.599862 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.599819    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.691110 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:07.691079    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331356f040de96b4d1948c4d6e1935b9.slice/crio-90dc91676c0324c61ff4fad3471dee35a0d907327c22b1775f67b91be558086f WatchSource:0}: Error finding container 90dc91676c0324c61ff4fad3471dee35a0d907327c22b1775f67b91be558086f: Status 404 returned error can't find the container with id 90dc91676c0324c61ff4fad3471dee35a0d907327c22b1775f67b91be558086f
Jul 10 02:40:07.700006 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.699887    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.700401 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.700387    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 02:40:07.712746 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:07.712719    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a70112095ce53725b3f8952b09deb60.slice/crio-0f55d29f9cf2ce4373b10f406b0b58f885005e63145693f1055ec6da46e2e23e WatchSource:0}: Error finding container 0f55d29f9cf2ce4373b10f406b0b58f885005e63145693f1055ec6da46e2e23e: Status 404 returned error can't find the container with id 0f55d29f9cf2ce4373b10f406b0b58f885005e63145693f1055ec6da46e2e23e
Jul 10 02:40:07.802240 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:07.802189    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-90.ec2.internal\" not found"
Jul 10 02:40:07.808184 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.808151    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:07.908055 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.907972    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:07.920161 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.920128    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 02:40:07.922043 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.922025    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal"
Jul 10 02:40:07.931163 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:07.931140    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 02:40:08.071224 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.071188    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:08.485199 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.485068    2551 apiserver.go:52] "Watching apiserver"
Jul 10 02:40:08.494575 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.494534    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 02:40:08.495649 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.495619    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mgjqj","openshift-multus/multus-additional-cni-plugins-g2gzv","openshift-ovn-kubernetes/ovnkube-node-vhtt6","kube-system/konnectivity-agent-l95bb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal","openshift-multus/multus-s2dk2","openshift-multus/network-metrics-daemon-5x97c","openshift-network-diagnostics/network-check-target-9rgvz","openshift-network-operator/iptables-alerter-zw5wr","kube-system/global-pull-secret-syncer-mg6bl","kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal","openshift-cluster-node-tuning-operator/tuned-54pnf","openshift-dns/node-resolver-7z8f6"]
Jul 10 02:40:08.498732 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.498709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.501722 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.501695    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 02:40:08.502326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.502128    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 02:40:08.502326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.502209    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cxs44\""
Jul 10 02:40:08.502326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.502241    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.502326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.502257    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.502584 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.502365    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 02:40:08.506041 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.505979    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.506430 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.506405    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.508504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.508434    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.508504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.508453    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-64knq\""
Jul 10 02:40:08.508504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.508498    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.508816 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.508797    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.508883 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.508829    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.509092 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.509067    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 02:40:08.509092 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.509090    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 02:40:08.509966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.509948    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 02:40:08.509966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.509961    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q46k4\""
Jul 10 02:40:08.510161 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.509999    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 02:40:08.510369 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.510340    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.511072 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.511053    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 02:40:08.511456 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.511437    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rgp28\""
Jul 10 02:40:08.511559 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.511461    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 02:40:08.512353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.511949    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.514426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.514426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514315    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cni-binary-copy\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.514426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514345    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-multus\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.514426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514374    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-multus-certs\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514423    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514473    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-conf\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514500    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-var-lib-kubelet\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514544    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-tmp\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-kubelet\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysconfig\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.514705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-os-release\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-netns\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514743    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-daemon-config\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514769    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-etc-kubernetes\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514795    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-tuned\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-system-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514851    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-bin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514881    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-hostroot\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514908    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjj6\" (UniqueName: \"kubernetes.io/projected/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-kube-api-access-ppjj6\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514932    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-modprobe-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.514975    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-run\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-systemd\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515035    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cnibin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515063    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-kubernetes\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515088    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-sys\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkmx\" (UniqueName: \"kubernetes.io/projected/bb669cd8-57a0-42de-9cc4-ceb85022e187-kube-api-access-lbkmx\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515123    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515152    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-socket-dir-parent\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515182    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-k8s-cni-cncf-io\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515289    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-conf-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515327    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-lib-modules\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515351    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-host\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.515776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.515773    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.516316 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.516047    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 02:40:08.516316 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.516060    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 02:40:08.516425 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.516327    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vwhx\""
Jul 10 02:40:08.517743 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.517720    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.521971 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.520646    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.521971 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.520803    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:08.521971 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.520891    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:08.521971 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.520921    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.521971 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.521622    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mgjg\""
Jul 10 02:40:08.526560 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.526524    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.529551 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.529441    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.531326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.530709    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.531326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.530924    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.531326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.530929    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j7qrb\""
Jul 10 02:40:08.531326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.531041    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 02:40:08.532088 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.532066    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 02:40:08.532318 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.532294    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 02:40:08.532534 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.532512    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnvpt\""
Jul 10 02:40:08.533581 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.533551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.535655 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.535633    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.535756 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.535727    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:08.536111 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.535931    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 02:40:08.536111 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.536105    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 02:40:08.536648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.536602    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2x9lw"
Jul 10 02:40:08.536737 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.536700    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z256g\""
Jul 10 02:40:08.536956 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.536936    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 02:40:08.538291 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.538270    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.538390 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.538340    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:08.546141 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.544302    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2x9lw"
Jul 10 02:40:08.602749 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.602719    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.615947    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-kubelet\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.615999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-tmp-dir\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616029    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-log-socket\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616055    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-env-overrides\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616074    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-kubelet\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-os-release\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616114    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-netns\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616150    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-netns\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616189    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-var-lib-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616219    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616239    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-os-release\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616248    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqsm\" (UniqueName: \"kubernetes.io/projected/9fa9e957-042c-44df-aa26-00bed3ab749f-kube-api-access-knqsm\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616306    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-netns\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616316    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-daemon-config\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.616383 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616381    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-systemd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-hostroot\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-modprobe-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616459    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-node-log\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616484    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616512    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-systemd\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616533    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-hostroot\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616539    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-systemd-units\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616573    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-cnibin\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616597    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-device-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616622    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-systemd\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cnibin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616662    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-kubernetes\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-sys\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616724    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-modprobe-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkmx\" (UniqueName: \"kubernetes.io/projected/bb669cd8-57a0-42de-9cc4-ceb85022e187-kube-api-access-lbkmx\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616759    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8h88\" (UniqueName: \"kubernetes.io/projected/9663ba75-8916-4780-96f8-ebb30457abd0-kube-api-access-n8h88\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.617379 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-kubernetes\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616786    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-kubelet\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616820    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-netd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616843    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616849    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-sys\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616860    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-hosts-file\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616884    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616912    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-system-cni-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9af8fba0-1650-4f7a-bc77-9b829352a916-agent-certs\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616960    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9af8fba0-1650-4f7a-bc77-9b829352a916-konnectivity-ca\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.616987    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617035    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617065    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-multus\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617093    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-conf\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-var-lib-kubelet\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617162    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-tmp\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617296    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cnibin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.618350 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617304    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617435    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-multus\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-conf\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-var-lib-kubelet\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617506    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9663ba75-8916-4780-96f8-ebb30457abd0-iptables-alerter-script\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617572    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtff\" (UniqueName: \"kubernetes.io/projected/0a266b18-49bd-41be-9055-71477697c3e1-kube-api-access-qqtff\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysconfig\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617623    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fa9e957-042c-44df-aa26-00bed3ab749f-ovn-node-metrics-cert\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617648    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-sys-fs\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617677    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mt5\" (UniqueName: \"kubernetes.io/projected/23bf4e90-9b65-407c-a3bd-8e87b7111503-kube-api-access-f4mt5\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617751    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-bin\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617837    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617848    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-config\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617863    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysconfig\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617888    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a266b18-49bd-41be-9055-71477697c3e1-host\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-daemon-config\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.619265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617915    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-os-release\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617956    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.617999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-etc-selinux\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618028    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-etc-kubernetes\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618076    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-tuned\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618086    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-etc-kubernetes\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618152    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-slash\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-ovn\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-dbus\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618310    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-system-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618360    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-bin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjj6\" (UniqueName: \"kubernetes.io/projected/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-kube-api-access-ppjj6\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-run\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618436    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-system-cni-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a266b18-49bd-41be-9055-71477697c3e1-serviceca\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.620139 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618475    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-kubelet-config\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618564    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-var-lib-cni-bin\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618601    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccsm\" (UniqueName: \"kubernetes.io/projected/d59d79d7-5041-45ff-90c2-9fb7ef380df9-kube-api-access-rccsm\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-registration-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618654    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7hp\" (UniqueName: \"kubernetes.io/projected/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-kube-api-access-2x7hp\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618680    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkfd\" (UniqueName: \"kubernetes.io/projected/43824797-43aa-4673-8294-a706183b1fc8-kube-api-access-lwkfd\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-socket-dir-parent\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618770    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-run\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-k8s-cni-cncf-io\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618849    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-conf-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618866    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-k8s-cni-cncf-io\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618879    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-lib-modules\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-socket-dir-parent\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618917    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-host\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9663ba75-8916-4780-96f8-ebb30457abd0-host-slash\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618917    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-multus-conf-dir\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618968    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-host\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.621967 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.618980    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-etc-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-socket-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-lib-modules\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cni-binary-copy\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619065    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-multus-certs\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619150    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-host-run-multus-certs\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619159    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-script-lib\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-sysctl-d\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.619550    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-cni-binary-copy\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.622909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.622103    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-tmp\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.623577 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.623207    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb669cd8-57a0-42de-9cc4-ceb85022e187-etc-tuned\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.626438 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.626368    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkmx\" (UniqueName: \"kubernetes.io/projected/bb669cd8-57a0-42de-9cc4-ceb85022e187-kube-api-access-lbkmx\") pod \"tuned-54pnf\" (UID: \"bb669cd8-57a0-42de-9cc4-ceb85022e187\") " pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.627262 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.627239    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjj6\" (UniqueName: \"kubernetes.io/projected/982771eb-e53e-4e98-ac09-4fe9cb1dd3dc-kube-api-access-ppjj6\") pod \"multus-s2dk2\" (UID: \"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc\") " pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.686287 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.686224    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal" event={"ID":"0a70112095ce53725b3f8952b09deb60","Type":"ContainerStarted","Data":"0f55d29f9cf2ce4373b10f406b0b58f885005e63145693f1055ec6da46e2e23e"}
Jul 10 02:40:08.687405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.687367    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal" event={"ID":"331356f040de96b4d1948c4d6e1935b9","Type":"ContainerStarted","Data":"90dc91676c0324c61ff4fad3471dee35a0d907327c22b1775f67b91be558086f"}
Jul 10 02:40:08.720311 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-slash\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720324    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-ovn\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720413    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-dbus\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720441    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a266b18-49bd-41be-9055-71477697c3e1-serviceca\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720467    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-kubelet-config\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccsm\" (UniqueName: \"kubernetes.io/projected/d59d79d7-5041-45ff-90c2-9fb7ef380df9-kube-api-access-rccsm\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.720520 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720521    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-registration-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720549    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7hp\" (UniqueName: \"kubernetes.io/projected/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-kube-api-access-2x7hp\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720576    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkfd\" (UniqueName: \"kubernetes.io/projected/43824797-43aa-4673-8294-a706183b1fc8-kube-api-access-lwkfd\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720608    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9663ba75-8916-4780-96f8-ebb30457abd0-host-slash\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720633    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-etc-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720659    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-socket-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720689    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-script-lib\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-tmp-dir\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-log-socket\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720791    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-env-overrides\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-registration-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720881    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-kubelet-config\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720928    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-slash\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.720980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.720960    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-socket-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721013    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6063745e-5b20-4c59-b4a0-3b7d34009b94-dbus\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721078    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-log-socket\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721089    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9663ba75-8916-4780-96f8-ebb30457abd0-host-slash\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721117    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-ovn\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721203    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-netns\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-var-lib-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721262    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqsm\" (UniqueName: \"kubernetes.io/projected/9fa9e957-042c-44df-aa26-00bed3ab749f-kube-api-access-knqsm\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721354    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-systemd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721380    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-node-log\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721405    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721438    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-systemd-units\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721464    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-cnibin\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721508    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-device-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.721790 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8h88\" (UniqueName: \"kubernetes.io/projected/9663ba75-8916-4780-96f8-ebb30457abd0-kube-api-access-n8h88\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721574    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-kubelet\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-netd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721617    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-var-lib-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721630    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-netns\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721708    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-kubelet\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721710    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-systemd-units\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721730    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-hosts-file\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721757    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-netd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721757    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-etc-openvswitch\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721764    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721730    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721798    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-system-cni-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721799    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-tmp-dir\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-system-cni-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721849    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-run-systemd\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.722750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9af8fba0-1650-4f7a-bc77-9b829352a916-agent-certs\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721915    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9af8fba0-1650-4f7a-bc77-9b829352a916-konnectivity-ca\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721921    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-device-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721922    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721966    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-node-log\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.721994    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-hosts-file\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722017    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722037    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9663ba75-8916-4780-96f8-ebb30457abd0-iptables-alerter-script\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722095    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtff\" (UniqueName: \"kubernetes.io/projected/0a266b18-49bd-41be-9055-71477697c3e1-kube-api-access-qqtff\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.722101    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.722156    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:09.222140619 +0000 UTC m=+3.035218694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722245    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722040    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722277    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-script-lib\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-env-overrides\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.723688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fa9e957-042c-44df-aa26-00bed3ab749f-ovn-node-metrics-cert\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722341    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-cnibin\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-sys-fs\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mt5\" (UniqueName: \"kubernetes.io/projected/23bf4e90-9b65-407c-a3bd-8e87b7111503-kube-api-access-f4mt5\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722490    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-bin\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722532    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-config\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722535    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9af8fba0-1650-4f7a-bc77-9b829352a916-konnectivity-ca\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722557    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a266b18-49bd-41be-9055-71477697c3e1-host\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722604    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-sys-fs\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722652    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-os-release\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722681    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-etc-selinux\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722710    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43824797-43aa-4673-8294-a706183b1fc8-os-release\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722746    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9663ba75-8916-4780-96f8-ebb30457abd0-iptables-alerter-script\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.722792    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a266b18-49bd-41be-9055-71477697c3e1-host\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.724604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722803    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43824797-43aa-4673-8294-a706183b1fc8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.722870    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:09.222850079 +0000 UTC m=+3.035928159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23bf4e90-9b65-407c-a3bd-8e87b7111503-etc-selinux\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.722986    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fa9e957-042c-44df-aa26-00bed3ab749f-host-cni-bin\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.723389    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9fa9e957-042c-44df-aa26-00bed3ab749f-ovnkube-config\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.723665    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a266b18-49bd-41be-9055-71477697c3e1-serviceca\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.725519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.725343    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fa9e957-042c-44df-aa26-00bed3ab749f-ovn-node-metrics-cert\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.725896 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.725606    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9af8fba0-1650-4f7a-bc77-9b829352a916-agent-certs\") pod \"konnectivity-agent-l95bb\" (UID: \"9af8fba0-1650-4f7a-bc77-9b829352a916\") " pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.730144 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.729433    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:08.730144 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.729464    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:08.730144 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.729481    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:08.730144 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:08.729548    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:09.229528469 +0000 UTC m=+3.042606558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:08.731796 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.731321    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccsm\" (UniqueName: \"kubernetes.io/projected/d59d79d7-5041-45ff-90c2-9fb7ef380df9-kube-api-access-rccsm\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:08.731796 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.731683    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkfd\" (UniqueName: \"kubernetes.io/projected/43824797-43aa-4673-8294-a706183b1fc8-kube-api-access-lwkfd\") pod \"multus-additional-cni-plugins-g2gzv\" (UID: \"43824797-43aa-4673-8294-a706183b1fc8\") " pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.733076 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.732106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7hp\" (UniqueName: \"kubernetes.io/projected/1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26-kube-api-access-2x7hp\") pod \"node-resolver-7z8f6\" (UID: \"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26\") " pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.733076 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.732986    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqsm\" (UniqueName: \"kubernetes.io/projected/9fa9e957-042c-44df-aa26-00bed3ab749f-kube-api-access-knqsm\") pod \"ovnkube-node-vhtt6\" (UID: \"9fa9e957-042c-44df-aa26-00bed3ab749f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.733486 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.733425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtff\" (UniqueName: \"kubernetes.io/projected/0a266b18-49bd-41be-9055-71477697c3e1-kube-api-access-qqtff\") pod \"node-ca-mgjqj\" (UID: \"0a266b18-49bd-41be-9055-71477697c3e1\") " pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.733841 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.733818    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8h88\" (UniqueName: \"kubernetes.io/projected/9663ba75-8916-4780-96f8-ebb30457abd0-kube-api-access-n8h88\") pod \"iptables-alerter-zw5wr\" (UID: \"9663ba75-8916-4780-96f8-ebb30457abd0\") " pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.734460 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.734416    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mt5\" (UniqueName: \"kubernetes.io/projected/23bf4e90-9b65-407c-a3bd-8e87b7111503-kube-api-access-f4mt5\") pod \"aws-ebs-csi-driver-node-chmg8\" (UID: \"23bf4e90-9b65-407c-a3bd-8e87b7111503\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.817030 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.816929    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2dk2"
Jul 10 02:40:08.829562 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.829409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54pnf"
Jul 10 02:40:08.830833 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:08.830729    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982771eb_e53e_4e98_ac09_4fe9cb1dd3dc.slice/crio-5b6662a5bdcda729d0ad2b391ffda6e62dee98e19b2a91289bd278e7e9df1430 WatchSource:0}: Error finding container 5b6662a5bdcda729d0ad2b391ffda6e62dee98e19b2a91289bd278e7e9df1430: Status 404 returned error can't find the container with id 5b6662a5bdcda729d0ad2b391ffda6e62dee98e19b2a91289bd278e7e9df1430
Jul 10 02:40:08.840296 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:08.840259    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb669cd8_57a0_42de_9cc4_ceb85022e187.slice/crio-04f2209c094adf77baf694167c6107195373fdda4f995f5ba36bf4849679f77f WatchSource:0}: Error finding container 04f2209c094adf77baf694167c6107195373fdda4f995f5ba36bf4849679f77f: Status 404 returned error can't find the container with id 04f2209c094adf77baf694167c6107195373fdda4f995f5ba36bf4849679f77f
Jul 10 02:40:08.841765 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.841726    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7z8f6"
Jul 10 02:40:08.852144 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:08.852110    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e33d7e9_3a83_4e47_8f5d_8ca4f4272a26.slice/crio-520f4d31c2c3f9efa7d5b0a20c51a302cd3da94eb6a92e9f2012572d56982060 WatchSource:0}: Error finding container 520f4d31c2c3f9efa7d5b0a20c51a302cd3da94eb6a92e9f2012572d56982060: Status 404 returned error can't find the container with id 520f4d31c2c3f9efa7d5b0a20c51a302cd3da94eb6a92e9f2012572d56982060
Jul 10 02:40:08.858825 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.858783    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:08.865269 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.865239    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:08.873490 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.873462    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8"
Jul 10 02:40:08.881478 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.881448    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgjqj"
Jul 10 02:40:08.891517 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.891487    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2gzv"
Jul 10 02:40:08.897905 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.897875    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zw5wr"
Jul 10 02:40:08.899305 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:08.899282    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:09.227200 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.227096    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:09.227200 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.227192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:09.227386 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.227213    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:09.227386 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.227288    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:10.22726865 +0000 UTC m=+4.040346747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:09.227386 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.227299    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:09.227386 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.227338    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:10.227325036 +0000 UTC m=+4.040403107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:09.287368 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.287334    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 02:40:09.328272 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.328220    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:09.328479 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.328389    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:09.328479 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.328412    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:09.328479 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.328425    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:09.328633 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:09.328494    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:10.328473212 +0000 UTC m=+4.141551282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:09.548273 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.548138    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:35:09 +0000 UTC" deadline="2028-04-20 04:37:57.709541324 +0000 UTC"
Jul 10 02:40:09.548273 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.548206    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15601h57m48.161340055s"
Jul 10 02:40:09.691373 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.691329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7z8f6" event={"ID":"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26","Type":"ContainerStarted","Data":"520f4d31c2c3f9efa7d5b0a20c51a302cd3da94eb6a92e9f2012572d56982060"}
Jul 10 02:40:09.693457 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.693411    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54pnf" event={"ID":"bb669cd8-57a0-42de-9cc4-ceb85022e187","Type":"ContainerStarted","Data":"04f2209c094adf77baf694167c6107195373fdda4f995f5ba36bf4849679f77f"}
Jul 10 02:40:09.694685 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:09.694655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2dk2" event={"ID":"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc","Type":"ContainerStarted","Data":"5b6662a5bdcda729d0ad2b391ffda6e62dee98e19b2a91289bd278e7e9df1430"}
Jul 10 02:40:10.099747 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:10.099703    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9663ba75_8916_4780_96f8_ebb30457abd0.slice/crio-bf6530f43e42ff871764ee743e3589944db933794d8cc0647a91d0afc7375961 WatchSource:0}: Error finding container bf6530f43e42ff871764ee743e3589944db933794d8cc0647a91d0afc7375961: Status 404 returned error can't find the container with id bf6530f43e42ff871764ee743e3589944db933794d8cc0647a91d0afc7375961
Jul 10 02:40:10.106412 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:10.106155    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa9e957_042c_44df_aa26_00bed3ab749f.slice/crio-00a704f89add0653157162b17298fc55ea4644a7cb5d40826f21cfb964c8fcd3 WatchSource:0}: Error finding container 00a704f89add0653157162b17298fc55ea4644a7cb5d40826f21cfb964c8fcd3: Status 404 returned error can't find the container with id 00a704f89add0653157162b17298fc55ea4644a7cb5d40826f21cfb964c8fcd3
Jul 10 02:40:10.108843 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:10.108678    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af8fba0_1650_4f7a_bc77_9b829352a916.slice/crio-d68d539042aed50455f9aff169444b9d95da47a1c569b8e2cfa97d4ee99b5f65 WatchSource:0}: Error finding container d68d539042aed50455f9aff169444b9d95da47a1c569b8e2cfa97d4ee99b5f65: Status 404 returned error can't find the container with id d68d539042aed50455f9aff169444b9d95da47a1c569b8e2cfa97d4ee99b5f65
Jul 10 02:40:10.237858 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.237756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:10.237858 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.237850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:10.238115 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.238042    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:10.238115 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.238116    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:12.238092356 +0000 UTC m=+6.051170435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:10.238281 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.238211    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:10.238335 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.238283    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:12.238261441 +0000 UTC m=+6.051339523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:10.338373 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.338331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:10.339017 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.338546    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:10.339017 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.338569    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:10.339017 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.338582    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:10.339017 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.338646    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:12.33862519 +0000 UTC m=+6.151703265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:10.548727 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.548623    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:35:09 +0000 UTC" deadline="2028-03-21 10:52:47.373941208 +0000 UTC"
Jul 10 02:40:10.548727 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.548667    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14888h12m36.825280994s"
Jul 10 02:40:10.682714 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.682407    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:10.682714 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.682418    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:10.682714 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.682553    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:10.682714 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.682577    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:10.682714 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.682709    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:10.683139 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:10.682799    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:10.697866 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.697827    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zw5wr" event={"ID":"9663ba75-8916-4780-96f8-ebb30457abd0","Type":"ContainerStarted","Data":"bf6530f43e42ff871764ee743e3589944db933794d8cc0647a91d0afc7375961"}
Jul 10 02:40:10.699313 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.699272    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8" event={"ID":"23bf4e90-9b65-407c-a3bd-8e87b7111503","Type":"ContainerStarted","Data":"c97af7b86aa618b90f7e593b76a71ebfd4be09f5c1fcb7036e83e1300d232627"}
Jul 10 02:40:10.700757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.700725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgjqj" event={"ID":"0a266b18-49bd-41be-9055-71477697c3e1","Type":"ContainerStarted","Data":"f6a99b943b8be21acca1735fd3cf18bef680cae1cb0cf324a83a2e768c0b9fb5"}
Jul 10 02:40:10.702164 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.702132    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l95bb" event={"ID":"9af8fba0-1650-4f7a-bc77-9b829352a916","Type":"ContainerStarted","Data":"d68d539042aed50455f9aff169444b9d95da47a1c569b8e2cfa97d4ee99b5f65"}
Jul 10 02:40:10.703645 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:10.703616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"00a704f89add0653157162b17298fc55ea4644a7cb5d40826f21cfb964c8fcd3"}
Jul 10 02:40:11.713231 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.712760    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal" event={"ID":"0a70112095ce53725b3f8952b09deb60","Type":"ContainerStarted","Data":"95d38d66129dcb014a82cdde3922aeb551b70725e22645fe4439cdbce798e05c"}
Jul 10 02:40:11.716662 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.716628    2551 generic.go:356] "Generic (PLEG): container finished" podID="331356f040de96b4d1948c4d6e1935b9" containerID="bf9e93d72fff6be7d014442452e45f4bb91fb405680848539ae17b1a8a2ae8b9" exitCode=0
Jul 10 02:40:11.716820 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.716721    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal" event={"ID":"331356f040de96b4d1948c4d6e1935b9","Type":"ContainerDied","Data":"bf9e93d72fff6be7d014442452e45f4bb91fb405680848539ae17b1a8a2ae8b9"}
Jul 10 02:40:11.723775 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.723261    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7z8f6" event={"ID":"1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26","Type":"ContainerStarted","Data":"c9c8ed42f079b035372247ca92be683d7a325524e231d3e4ede9beb9de3fe683"}
Jul 10 02:40:11.732639 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.730936    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-90.ec2.internal" podStartSLOduration=3.73091527 podStartE2EDuration="3.73091527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:11.730081166 +0000 UTC m=+5.543159262" watchObservedRunningTime="2026-07-10 02:40:11.73091527 +0000 UTC m=+5.543993368"
Jul 10 02:40:11.734089 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.733249    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerStarted","Data":"6408db98b216599343b7d67fdb0dd6a9673854744ac384a88fef2547b0dacf71"}
Jul 10 02:40:11.746915 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:11.746004    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7z8f6" podStartSLOduration=2.557133608 podStartE2EDuration="4.745981137s" totalImagesPullingTime="2.188847529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:11.74538201 +0000 UTC m=+5.558460105" watchObservedRunningTime="2026-07-10 02:40:11.745981137 +0000 UTC m=+5.559059233"
Jul 10 02:40:12.256660 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.256563    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:12.256660 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.256656    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:12.256905 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.256811    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:12.256905 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.256882    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:16.256858971 +0000 UTC m=+10.069937060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:12.257401 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.257378    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:12.257513 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.257445    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:16.257428062 +0000 UTC m=+10.070506136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:12.358251 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.358161    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:12.358432 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.358361    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:12.358432 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.358388    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:12.358432 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.358404    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:12.358615 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.358471    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:16.358449185 +0000 UTC m=+10.171527259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:12.682287 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.682250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:12.682468 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.682301    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:12.682468 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.682400    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:12.682468 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.682467    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:12.682659 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.682586    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:12.682717 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:12.682681    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:12.751698 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:12.751659    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal" event={"ID":"331356f040de96b4d1948c4d6e1935b9","Type":"ContainerStarted","Data":"f679351ba0a6ad5b216af92f118a7d2b7d0b390ed42fa13310d9ae8dece423f5"}
Jul 10 02:40:13.767701 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:13.767664    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zw5wr" event={"ID":"9663ba75-8916-4780-96f8-ebb30457abd0","Type":"ContainerStarted","Data":"0ea57d16d05870c1ad5d579b80f480fa145d8c050af4bc4ec39d2206255e22a5"}
Jul 10 02:40:13.786880 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:13.786808    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-90.ec2.internal" podStartSLOduration=5.786783283 podStartE2EDuration="5.786783283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:12.770297109 +0000 UTC m=+6.583375205" watchObservedRunningTime="2026-07-10 02:40:13.786783283 +0000 UTC m=+7.599861379"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:14.682479    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:14.682628    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:14.682670    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:14.682708    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:14.682847    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:14.683048 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:14.682959    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:16.293675 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.293629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:16.294168 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.293713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:16.294168 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.293788    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:16.294168 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.293814    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:16.294168 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.293865    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:24.293844885 +0000 UTC m=+18.106922957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:16.294168 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.293882    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:24.293873002 +0000 UTC m=+18.106951072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:16.394373 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.394206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:16.394572 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.394432    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:16.394572 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.394457    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:16.394572 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.394470    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:16.394572 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.394544    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:24.394523401 +0000 UTC m=+18.207601487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:16.682725 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.682511    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:16.682725 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.682648    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:16.682725 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.682691    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:16.683035 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.682767    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:16.683035 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:16.682811    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:16.683035 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:16.682893    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:18.681778    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:18.681932    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:18.681944    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:18.682071    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:18.682154    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:18.682385 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:18.682273    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:20.681720 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:20.681681    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:20.682187 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:20.681727    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:20.682187 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:20.681818    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:20.682187 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:20.681862    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:20.682187 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:20.681969    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:20.682187 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:20.682030    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:22.682218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:22.682180    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:22.682683 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:22.682180    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:22.682683 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:22.682312    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:22.682683 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:22.682334    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:22.682683 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:22.682443    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:22.682683 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:22.682544    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:24.357336 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.357258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:24.357336 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.357319    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:24.357903 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.357407    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:24.357903 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.357425    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:24.357903 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.357485    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:40.35746755 +0000 UTC m=+34.170545626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:24.357903 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.357505    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:40.357493594 +0000 UTC m=+34.170571663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:24.458271 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.458237    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:24.458472 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.458410    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:24.458472 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.458430    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:24.458472 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.458442    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:24.458634 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.458511    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:40:40.458491006 +0000 UTC m=+34.271569091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:24.681776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.681688    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:24.681942 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.681688    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:24.681942 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.681825    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:24.681942 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:24.681703    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:24.681942 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.681909    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:24.682159 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:24.682003    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:26.682794 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:26.682727    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:26.683334 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:26.682848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:26.683334 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:26.682862    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:26.683334 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:26.682968    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:26.683334 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:26.683036    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:26.683334 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:26.683123    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.682375    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.682398    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:28.682520    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.682544    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:28.682654    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:28.682891 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:28.682739    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:28.809647 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.809409    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerStarted","Data":"5a62b41d9fb04a63bf7e48753a6fda35241da3e1d37fbbeb533fd3214a91da7e"}
Jul 10 02:40:28.811472 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.811441    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l95bb" event={"ID":"9af8fba0-1650-4f7a-bc77-9b829352a916","Type":"ContainerStarted","Data":"1fe29075dd49c76a7289c496021154f94d89a3caeb50de87247fa2d8c58303e1"}
Jul 10 02:40:28.834658 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.834609    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zw5wr" podStartSLOduration=20.898568231 podStartE2EDuration="21.834593774s" totalImagesPullingTime="936.025543ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:13.786163207 +0000 UTC m=+7.599241303" watchObservedRunningTime="2026-07-10 02:40:28.834593774 +0000 UTC m=+22.647671864"
Jul 10 02:40:28.850496 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:28.850452    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l95bb" podStartSLOduration=3.3680323039999998 podStartE2EDuration="21.85043481s" totalImagesPullingTime="18.482402506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:28.850405051 +0000 UTC m=+22.663483144" watchObservedRunningTime="2026-07-10 02:40:28.85043481 +0000 UTC m=+22.663512895"
Jul 10 02:40:29.814914 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.814877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54pnf" event={"ID":"bb669cd8-57a0-42de-9cc4-ceb85022e187","Type":"ContainerStarted","Data":"4b52184084ab2bb4bf341e6ba341c34ad6e7f78809957f31e057182050154a2f"}
Jul 10 02:40:29.816409 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.816379    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2dk2" event={"ID":"982771eb-e53e-4e98-ac09-4fe9cb1dd3dc","Type":"ContainerStarted","Data":"d8f7344150640a51566877ad86ab83f3fe1b063849f00620cd2fc43615cf70ac"}
Jul 10 02:40:29.817746 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.817719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8" event={"ID":"23bf4e90-9b65-407c-a3bd-8e87b7111503","Type":"ContainerStarted","Data":"4196432faee513f38677f1d9378bf86e913ed54265ce8eb4598d03d214338027"}
Jul 10 02:40:29.818889 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.818870    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgjqj" event={"ID":"0a266b18-49bd-41be-9055-71477697c3e1","Type":"ContainerStarted","Data":"db95759bba23e4c6f2d38f116e69ebadb891346da01157539147ced23d9f81bc"}
Jul 10 02:40:29.820364 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.820335    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="5a62b41d9fb04a63bf7e48753a6fda35241da3e1d37fbbeb533fd3214a91da7e" exitCode=0
Jul 10 02:40:29.820460 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.820365    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"5a62b41d9fb04a63bf7e48753a6fda35241da3e1d37fbbeb533fd3214a91da7e"}
Jul 10 02:40:29.823024 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823009    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:40:29.823325 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823308    2551 generic.go:356] "Generic (PLEG): container finished" podID="9fa9e957-042c-44df-aa26-00bed3ab749f" containerID="1e06aa18ad2ff39837cbc490b34576b6009cebdaa9047788ca6196e0ca21a898" exitCode=1
Jul 10 02:40:29.823391 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"3a5343721cbd563added39472ad1f5acb49797405672a61e340ce9339e884b06"}
Jul 10 02:40:29.823458 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823445    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"bf46a14a52ae1dffac7ffab88ba3841444d9961cc28ef98806532b7627563314"}
Jul 10 02:40:29.823510 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"da45d8bb03f3ec1f1c70f33db451db058739e14561a93d9b42ffab548cd83a37"}
Jul 10 02:40:29.823563 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823550    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"adfba6c4b4d916eb6b1c531cbadc3b066338d7224f853116d9a6eede696c36e8"}
Jul 10 02:40:29.823612 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerDied","Data":"1e06aa18ad2ff39837cbc490b34576b6009cebdaa9047788ca6196e0ca21a898"}
Jul 10 02:40:29.823667 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.823655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"dc37bafeae50ee266343ad55bf33e7a0baadbd16824ed96ec46d8121a25056fd"}
Jul 10 02:40:29.831578 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.831543    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-54pnf" podStartSLOduration=3.081659943 podStartE2EDuration="22.831530826s" totalImagesPullingTime="19.749870883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:29.831331216 +0000 UTC m=+23.644409308" watchObservedRunningTime="2026-07-10 02:40:29.831530826 +0000 UTC m=+23.644608917"
Jul 10 02:40:29.845272 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:29.845230    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mgjqj" podStartSLOduration=4.530327143 podStartE2EDuration="22.845215784s" totalImagesPullingTime="18.314888641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:29.84472627 +0000 UTC m=+23.657804360" watchObservedRunningTime="2026-07-10 02:40:29.845215784 +0000 UTC m=+23.658293875"
Jul 10 02:40:30.058725 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.058557    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 02:40:30.581185 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.581063    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T02:40:30.058716993Z","UUID":"37c90301-61ac-40b6-b775-9bf64e12f5f4","Handler":null,"Name":"","Endpoint":""}
Jul 10 02:40:30.582640 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.582619    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 02:40:30.582761 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.582655    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 02:40:30.681631 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.681596    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:30.681817 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.681596    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:30.681817 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:30.681723    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:30.681817 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.681596    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:30.682035 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:30.681769    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:30.682035 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:30.681889    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:30.827658 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:30.827625    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8" event={"ID":"23bf4e90-9b65-407c-a3bd-8e87b7111503","Type":"ContainerStarted","Data":"847689f02343d5f8eea744888bb4e9aa93f54666921520b1e5f0dcd2e1723830"}
Jul 10 02:40:31.835798 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.835715    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:40:31.836396 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.836230    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"f218596ceacb67713687e84558255afadefa514aa67d7ae0069e397926d116e9"}
Jul 10 02:40:31.838719 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.838693    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8" event={"ID":"23bf4e90-9b65-407c-a3bd-8e87b7111503","Type":"ContainerStarted","Data":"0bda4ce92b66ef06d8cd44ea1aa29a45c5e78ad238ae28b4bd24fb7c112f7f57"}
Jul 10 02:40:31.856897 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.856850    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chmg8" podStartSLOduration=4.072814891 podStartE2EDuration="24.856829922s" totalImagesPullingTime="20.784015031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:31.856512181 +0000 UTC m=+25.669590273" watchObservedRunningTime="2026-07-10 02:40:31.856829922 +0000 UTC m=+25.669908015"
Jul 10 02:40:31.857087 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.856979    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s2dk2" podStartSLOduration=5.074151015 podStartE2EDuration="24.856970979s" totalImagesPullingTime="19.782819964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:29.889717832 +0000 UTC m=+23.702795924" watchObservedRunningTime="2026-07-10 02:40:31.856970979 +0000 UTC m=+25.670049070"
Jul 10 02:40:31.894668 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.894624    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:31.895343 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:31.895322    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:32.681838 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:32.681803    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:32.681838 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:32.681824    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:32.682043 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:32.681803    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:32.682043 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:32.681918    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:32.682043 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:32.682022    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:32.682166 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:32.682114    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:34.681786 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.681710    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:34.681786 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.681737    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:34.682457 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.681710    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:34.682457 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:34.681816    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:34.682457 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:34.681889    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:34.682457 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:34.681966    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:34.848957 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.848771    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:40:34.849353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.849320    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"f72802871eeca9e4cbd29ce492dd63cdc81c37f3ae9b5751cdc27487dc8e3c7b"}
Jul 10 02:40:34.849640 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.849621    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:34.849787 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.849770    2551 scope.go:117] "RemoveContainer" containerID="1e06aa18ad2ff39837cbc490b34576b6009cebdaa9047788ca6196e0ca21a898"
Jul 10 02:40:34.868639 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:34.868617    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:35.854286 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.854193    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="f076e37970be5ae13707451c9db369882a9de68bf8a0b56a3ee35fdd5714646f" exitCode=0
Jul 10 02:40:35.854755 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.854296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"f076e37970be5ae13707451c9db369882a9de68bf8a0b56a3ee35fdd5714646f"}
Jul 10 02:40:35.866980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.865348    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:40:35.866980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.865725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" event={"ID":"9fa9e957-042c-44df-aa26-00bed3ab749f","Type":"ContainerStarted","Data":"62e341289cd8c96fc37b185d882b90cc9d34eea5b73341710bb242673b2a142d"}
Jul 10 02:40:35.866980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.866796    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:35.866980 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.866935    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:35.888316 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.888290    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:40:35.909899 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:35.909851    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6" podStartSLOduration=10.231396863 podStartE2EDuration="28.909834719s" totalImagesPullingTime="18.678437856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:35.90836362 +0000 UTC m=+29.721441723" watchObservedRunningTime="2026-07-10 02:40:35.909834719 +0000 UTC m=+29.722912811"
Jul 10 02:40:36.538678 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.538645    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mg6bl"]
Jul 10 02:40:36.539458 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.538818    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:36.539458 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:36.538969    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:36.539458 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.539286    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rgvz"]
Jul 10 02:40:36.539458 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.539398    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:36.539672 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:36.539494    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:36.541086 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.541061    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5x97c"]
Jul 10 02:40:36.541244 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:36.541177    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:36.541310 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:36.541264    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:37.682310 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:37.682280    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:37.682688 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:37.682388    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:37.682688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:37.682419    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:37.682688 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:37.682476    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:37.873201 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:37.872997    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="59e9fe2142a3e438ab54c79652dc401071fee79d5d041f92bea8645dc16c158b" exitCode=0
Jul 10 02:40:37.873360 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:37.873080    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"59e9fe2142a3e438ab54c79652dc401071fee79d5d041f92bea8645dc16c158b"}
Jul 10 02:40:38.036951 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.036880    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:38.037111 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.037013    2551 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 10 02:40:38.037613 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.037591    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l95bb"
Jul 10 02:40:38.681969 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.681933    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:38.682257 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:38.682082    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:38.878217 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.878180    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="c7f8faf81774584332def43ad04328d63b44f69b3d790090f6285602f236fab3" exitCode=0
Jul 10 02:40:38.878659 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:38.878266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"c7f8faf81774584332def43ad04328d63b44f69b3d790090f6285602f236fab3"}
Jul 10 02:40:39.099277 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:39.099210    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7z8f6_1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26/dns-node-resolver/0.log"
Jul 10 02:40:39.682175 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:39.682126    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:39.682175 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:39.682158    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:39.682406 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:39.682268    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5x97c" podUID="d59d79d7-5041-45ff-90c2-9fb7ef380df9"
Jul 10 02:40:39.682463 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:39.682402    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rgvz" podUID="9a53e3f4-b9e5-46f5-a87a-8e73be826cfb"
Jul 10 02:40:40.270485 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:40.270457    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgjqj_0a266b18-49bd-41be-9055-71477697c3e1/node-ca/0.log"
Jul 10 02:40:40.380022 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:40.379974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:40.380022 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:40.380036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:40.380255 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.380139    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:40.380255 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.380209    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret podName:6063745e-5b20-4c59-b4a0-3b7d34009b94 nodeName:}" failed. No retries permitted until 2026-07-10 02:41:12.380193659 +0000 UTC m=+66.193271729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret") pod "global-pull-secret-syncer-mg6bl" (UID: "6063745e-5b20-4c59-b4a0-3b7d34009b94") : object "kube-system"/"original-pull-secret" not registered
Jul 10 02:40:40.380255 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.380139    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:40.380358 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.380276    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs podName:d59d79d7-5041-45ff-90c2-9fb7ef380df9 nodeName:}" failed. No retries permitted until 2026-07-10 02:41:12.380264313 +0000 UTC m=+66.193342386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs") pod "network-metrics-daemon-5x97c" (UID: "d59d79d7-5041-45ff-90c2-9fb7ef380df9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 02:40:40.480565 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:40.480525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:40.480751 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.480713    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 02:40:40.480751 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.480736    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 02:40:40.480751 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.480747    2551 projected.go:196] Error preparing data for projected volume kube-api-access-p4f2j for pod openshift-network-diagnostics/network-check-target-9rgvz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:40.480893 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.480803    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j podName:9a53e3f4-b9e5-46f5-a87a-8e73be826cfb nodeName:}" failed. No retries permitted until 2026-07-10 02:41:12.480785864 +0000 UTC m=+66.293863949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4f2j" (UniqueName: "kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j") pod "network-check-target-9rgvz" (UID: "9a53e3f4-b9e5-46f5-a87a-8e73be826cfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 02:40:40.682122 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:40.682024    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:40.682273 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:40.682156    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg6bl" podUID="6063745e-5b20-4c59-b4a0-3b7d34009b94"
Jul 10 02:40:41.483635 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.483555    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-90.ec2.internal" event="NodeReady"
Jul 10 02:40:41.484058 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.484045    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 02:40:41.531868 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.531823    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dn7g5"]
Jul 10 02:40:41.536050 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.536027    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.539127 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.538977    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 02:40:41.539782 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.539581    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 02:40:41.539782 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.539593    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6649n"]
Jul 10 02:40:41.539782 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.539608    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l52l\""
Jul 10 02:40:41.544272 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.544246    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.545851 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.545826    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dn7g5"]
Jul 10 02:40:41.547092 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.546915    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 02:40:41.547092 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.546980    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2tgb9\""
Jul 10 02:40:41.547276 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.547139    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 02:40:41.547276 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.547199    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 02:40:41.552502 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.552432    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6649n"]
Jul 10 02:40:41.559519 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.559497    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qtm94"]
Jul 10 02:40:41.563436 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.563413    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.566804 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.566780    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 02:40:41.566804 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.566806    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 02:40:41.566993 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.566838    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 02:40:41.566993 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.566878    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c8l6g\""
Jul 10 02:40:41.566993 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.566908    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 02:40:41.574429 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.574405    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qtm94"]
Jul 10 02:40:41.682215 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.682184    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:40:41.682215 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.682197    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:40:41.684902 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.684880    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 02:40:41.684902 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.684880    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 02:40:41.685126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.684881    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 02:40:41.685126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.685036    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jw76v\""
Jul 10 02:40:41.685126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.685056    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h9pgv\""
Jul 10 02:40:41.688007 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.687990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28f636a3-81c8-4492-a301-98f47fdc30d3-crio-socket\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.688110 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688035    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ac0f6-871a-45ea-8330-d1d901a072b9-cert\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.688110 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688056    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bzb\" (UniqueName: \"kubernetes.io/projected/0e8ac0f6-871a-45ea-8330-d1d901a072b9-kube-api-access-h9bzb\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.688218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688136    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b282j\" (UniqueName: \"kubernetes.io/projected/1018b000-e21e-4f77-806f-da744fa85e52-kube-api-access-b282j\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.688218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688168    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1018b000-e21e-4f77-806f-da744fa85e52-config-volume\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.688218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.688218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688213    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018b000-e21e-4f77-806f-da744fa85e52-tmp-dir\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.688353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688233    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.688353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688250    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktnz\" (UniqueName: \"kubernetes.io/projected/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-api-access-hktnz\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.688353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.688353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.688313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f636a3-81c8-4492-a301-98f47fdc30d3-data-volume\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789451 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28f636a3-81c8-4492-a301-98f47fdc30d3-crio-socket\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789451 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789428    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ac0f6-871a-45ea-8330-d1d901a072b9-cert\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789460    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bzb\" (UniqueName: \"kubernetes.io/projected/0e8ac0f6-871a-45ea-8330-d1d901a072b9-kube-api-access-h9bzb\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b282j\" (UniqueName: \"kubernetes.io/projected/1018b000-e21e-4f77-806f-da744fa85e52-kube-api-access-b282j\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1018b000-e21e-4f77-806f-da744fa85e52-config-volume\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789573    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789602    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018b000-e21e-4f77-806f-da744fa85e52-tmp-dir\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789631    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789642    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/28f636a3-81c8-4492-a301-98f47fdc30d3-crio-socket\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789655    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktnz\" (UniqueName: \"kubernetes.io/projected/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-api-access-hktnz\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789697    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.789966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f636a3-81c8-4492-a301-98f47fdc30d3-data-volume\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.789966 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:41.789898    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 02:40:41.789966 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:41.789958    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls podName:1018b000-e21e-4f77-806f-da744fa85e52 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:42.289938993 +0000 UTC m=+36.103017085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls") pod "dns-default-dn7g5" (UID: "1018b000-e21e-4f77-806f-da744fa85e52") : secret "dns-default-metrics-tls" not found
Jul 10 02:40:41.790318 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:41.789998    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:41.790318 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.789999    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1018b000-e21e-4f77-806f-da744fa85e52-tmp-dir\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.790318 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:41.790057    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls podName:28f636a3-81c8-4492-a301-98f47fdc30d3 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:42.290040825 +0000 UTC m=+36.103118919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qtm94" (UID: "28f636a3-81c8-4492-a301-98f47fdc30d3") : secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:41.790318 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.790136    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f636a3-81c8-4492-a301-98f47fdc30d3-data-volume\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.790498 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.790468    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1018b000-e21e-4f77-806f-da744fa85e52-config-volume\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.790548 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.790506    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.794305 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.794285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ac0f6-871a-45ea-8330-d1d901a072b9-cert\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.804901 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.804869    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktnz\" (UniqueName: \"kubernetes.io/projected/28f636a3-81c8-4492-a301-98f47fdc30d3-kube-api-access-hktnz\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:41.805066 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.804871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b282j\" (UniqueName: \"kubernetes.io/projected/1018b000-e21e-4f77-806f-da744fa85e52-kube-api-access-b282j\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:41.805066 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.805026    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bzb\" (UniqueName: \"kubernetes.io/projected/0e8ac0f6-871a-45ea-8330-d1d901a072b9-kube-api-access-h9bzb\") pod \"ingress-canary-6649n\" (UID: \"0e8ac0f6-871a-45ea-8330-d1d901a072b9\") " pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:41.856087 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:41.856051    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6649n"
Jul 10 02:40:42.012345 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:42.012153    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6649n"]
Jul 10 02:40:42.293718 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:42.293671    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:42.293718 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:42.293722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:42.293991 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:42.293805    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 02:40:42.293991 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:42.293862    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls podName:1018b000-e21e-4f77-806f-da744fa85e52 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:43.293847611 +0000 UTC m=+37.106925680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls") pod "dns-default-dn7g5" (UID: "1018b000-e21e-4f77-806f-da744fa85e52") : secret "dns-default-metrics-tls" not found
Jul 10 02:40:42.293991 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:42.293805    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:42.293991 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:42.293902    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls podName:28f636a3-81c8-4492-a301-98f47fdc30d3 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:43.293893611 +0000 UTC m=+37.106971683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qtm94" (UID: "28f636a3-81c8-4492-a301-98f47fdc30d3") : secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:42.682241 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:42.682148    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:40:42.684946 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:42.684920    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 02:40:43.113564 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.113467    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hjmk4"]
Jul 10 02:40:43.117858 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.117827    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.120375 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.120345    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 10 02:40:43.120502 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.120345    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 10 02:40:43.121769 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.121636    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rq5nt\""
Jul 10 02:40:43.121769 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.121650    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 10 02:40:43.121769 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.121661    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 10 02:40:43.125606 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.125481    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hjmk4"]
Jul 10 02:40:43.202557 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.202509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-key\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.202750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.202574    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmvr\" (UniqueName: \"kubernetes.io/projected/b0619e0a-d502-4a34-9a50-c29094535fa3-kube-api-access-zrmvr\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.202750 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.202668    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-cabundle\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.304023 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.303980    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-key\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.304023 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.304035    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmvr\" (UniqueName: \"kubernetes.io/projected/b0619e0a-d502-4a34-9a50-c29094535fa3-kube-api-access-zrmvr\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.304295 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.304110    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-cabundle\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.304295 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.304183    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:43.304295 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.304206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:43.304448 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:43.304308    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 02:40:43.304448 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:43.304338    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:43.304448 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:43.304369    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls podName:1018b000-e21e-4f77-806f-da744fa85e52 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:45.304349432 +0000 UTC m=+39.117427501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls") pod "dns-default-dn7g5" (UID: "1018b000-e21e-4f77-806f-da744fa85e52") : secret "dns-default-metrics-tls" not found
Jul 10 02:40:43.304448 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:43.304408    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls podName:28f636a3-81c8-4492-a301-98f47fdc30d3 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:45.304389462 +0000 UTC m=+39.117467537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qtm94" (UID: "28f636a3-81c8-4492-a301-98f47fdc30d3") : secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:43.304918 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.304891    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-cabundle\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.307787 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.307766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b0619e0a-d502-4a34-9a50-c29094535fa3-signing-key\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.312280 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.312258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmvr\" (UniqueName: \"kubernetes.io/projected/b0619e0a-d502-4a34-9a50-c29094535fa3-kube-api-access-zrmvr\") pod \"service-ca-79f68b9f4f-hjmk4\" (UID: \"b0619e0a-d502-4a34-9a50-c29094535fa3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.429361 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.429281    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4"
Jul 10 02:40:43.476167 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:43.476118    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ac0f6_871a_45ea_8330_d1d901a072b9.slice/crio-359563b985062e2c4baed29e0615c2a87a3e87560da757a3650d4595388c16c7 WatchSource:0}: Error finding container 359563b985062e2c4baed29e0615c2a87a3e87560da757a3650d4595388c16c7: Status 404 returned error can't find the container with id 359563b985062e2c4baed29e0615c2a87a3e87560da757a3650d4595388c16c7
Jul 10 02:40:43.608230 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.607975    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-hjmk4"]
Jul 10 02:40:43.612693 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:43.612668    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0619e0a_d502_4a34_9a50_c29094535fa3.slice/crio-ab072225c571c10ac355f60cc133a8f34c139cd90791a6cef42e90dc97aaca2a WatchSource:0}: Error finding container ab072225c571c10ac355f60cc133a8f34c139cd90791a6cef42e90dc97aaca2a: Status 404 returned error can't find the container with id ab072225c571c10ac355f60cc133a8f34c139cd90791a6cef42e90dc97aaca2a
Jul 10 02:40:43.892818 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.892781    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="4ee33aeff70947abfc68f511008df871ba41d9be8cb77aa6512ab60f167c787a" exitCode=0
Jul 10 02:40:43.892818 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.892802    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"4ee33aeff70947abfc68f511008df871ba41d9be8cb77aa6512ab60f167c787a"}
Jul 10 02:40:43.894158 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.894133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4" event={"ID":"b0619e0a-d502-4a34-9a50-c29094535fa3","Type":"ContainerStarted","Data":"ab072225c571c10ac355f60cc133a8f34c139cd90791a6cef42e90dc97aaca2a"}
Jul 10 02:40:43.895173 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:43.895145    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6649n" event={"ID":"0e8ac0f6-871a-45ea-8330-d1d901a072b9","Type":"ContainerStarted","Data":"359563b985062e2c4baed29e0615c2a87a3e87560da757a3650d4595388c16c7"}
Jul 10 02:40:44.902185 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:44.902147    2551 generic.go:356] "Generic (PLEG): container finished" podID="43824797-43aa-4673-8294-a706183b1fc8" containerID="e28aed2a24e6ebe0753522b7748621d090b979f19613217b85575fdc24893769" exitCode=0
Jul 10 02:40:44.902687 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:44.902196    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerDied","Data":"e28aed2a24e6ebe0753522b7748621d090b979f19613217b85575fdc24893769"}
Jul 10 02:40:45.320734 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:45.320451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:45.320734 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:45.320708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:45.320941 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:45.320544    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:45.320941 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:45.320815    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls podName:28f636a3-81c8-4492-a301-98f47fdc30d3 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:49.320796638 +0000 UTC m=+43.133874709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qtm94" (UID: "28f636a3-81c8-4492-a301-98f47fdc30d3") : secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:45.320941 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:45.320824    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 02:40:45.320941 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:45.320871    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls podName:1018b000-e21e-4f77-806f-da744fa85e52 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:49.320854148 +0000 UTC m=+43.133932231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls") pod "dns-default-dn7g5" (UID: "1018b000-e21e-4f77-806f-da744fa85e52") : secret "dns-default-metrics-tls" not found
Jul 10 02:40:45.909220 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:45.909111    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" event={"ID":"43824797-43aa-4673-8294-a706183b1fc8","Type":"ContainerStarted","Data":"24fb436bf5e2475206588689b5f083e5db2b3f0e67e47a55fa2c55dd709197b6"}
Jul 10 02:40:45.934415 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:45.934350    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2gzv" podStartSLOduration=10.01913273 podStartE2EDuration="38.934331112s" totalImagesPullingTime="28.915198382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:45.932706378 +0000 UTC m=+39.745784470" watchObservedRunningTime="2026-07-10 02:40:45.934331112 +0000 UTC m=+39.747409206"
Jul 10 02:40:47.916466 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:47.916424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4" event={"ID":"b0619e0a-d502-4a34-9a50-c29094535fa3","Type":"ContainerStarted","Data":"1370d6ce4627bf3e9a01c8fc4e2f2289d9b83e2311fe2f23a85131ae429cd5cd"}
Jul 10 02:40:47.917972 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:47.917949    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6649n" event={"ID":"0e8ac0f6-871a-45ea-8330-d1d901a072b9","Type":"ContainerStarted","Data":"c4b5c012a69235e4ce270b09cc29b28ba653eebbe6364a3db7273e78877385f6"}
Jul 10 02:40:47.949543 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:47.949467    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6649n" podStartSLOduration=3.491949217 podStartE2EDuration="6.949448397s" totalImagesPullingTime="3.45749918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:47.948835062 +0000 UTC m=+41.761913153" watchObservedRunningTime="2026-07-10 02:40:47.949448397 +0000 UTC m=+41.762526487"
Jul 10 02:40:47.949688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:47.949550    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-hjmk4" podStartSLOduration=1.622986939 podStartE2EDuration="4.949544386s" totalImagesPullingTime="3.326557447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:40:47.932537733 +0000 UTC m=+41.745615825" watchObservedRunningTime="2026-07-10 02:40:47.949544386 +0000 UTC m=+41.762622477"
Jul 10 02:40:49.355683 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:49.355638    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:49.356046 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:49.355765    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:49.356046 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:49.355801    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 02:40:49.356046 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:49.355847    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:49.356046 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:49.355877    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls podName:1018b000-e21e-4f77-806f-da744fa85e52 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:57.355859431 +0000 UTC m=+51.168937515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls") pod "dns-default-dn7g5" (UID: "1018b000-e21e-4f77-806f-da744fa85e52") : secret "dns-default-metrics-tls" not found
Jul 10 02:40:49.356046 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:40:49.355893    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls podName:28f636a3-81c8-4492-a301-98f47fdc30d3 nodeName:}" failed. No retries permitted until 2026-07-10 02:40:57.355886984 +0000 UTC m=+51.168965054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qtm94" (UID: "28f636a3-81c8-4492-a301-98f47fdc30d3") : secret "insights-runtime-extractor-tls" not found
Jul 10 02:40:57.416022 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.415968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:57.416022 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.416030    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:57.419704 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.419674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/28f636a3-81c8-4492-a301-98f47fdc30d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qtm94\" (UID: \"28f636a3-81c8-4492-a301-98f47fdc30d3\") " pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:57.419704 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.419683    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1018b000-e21e-4f77-806f-da744fa85e52-metrics-tls\") pod \"dns-default-dn7g5\" (UID: \"1018b000-e21e-4f77-806f-da744fa85e52\") " pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:57.448599 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.448562    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:40:57.473679 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.473649    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qtm94"
Jul 10 02:40:57.635515 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.635480    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dn7g5"]
Jul 10 02:40:57.639334 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:57.639304    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1018b000_e21e_4f77_806f_da744fa85e52.slice/crio-69d23f59d61d07ae87455355b675d3aefe440176213805ddc32a89fd62898907 WatchSource:0}: Error finding container 69d23f59d61d07ae87455355b675d3aefe440176213805ddc32a89fd62898907: Status 404 returned error can't find the container with id 69d23f59d61d07ae87455355b675d3aefe440176213805ddc32a89fd62898907
Jul 10 02:40:57.652700 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.652673    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qtm94"]
Jul 10 02:40:57.655826 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:40:57.655792    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f636a3_81c8_4492_a301_98f47fdc30d3.slice/crio-b2bdb80c08654b556d823bb7116a21d3cc960a0bfda10b8bb0ef098432135e65 WatchSource:0}: Error finding container b2bdb80c08654b556d823bb7116a21d3cc960a0bfda10b8bb0ef098432135e65: Status 404 returned error can't find the container with id b2bdb80c08654b556d823bb7116a21d3cc960a0bfda10b8bb0ef098432135e65
Jul 10 02:40:57.946382 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.946344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn7g5" event={"ID":"1018b000-e21e-4f77-806f-da744fa85e52","Type":"ContainerStarted","Data":"69d23f59d61d07ae87455355b675d3aefe440176213805ddc32a89fd62898907"}
Jul 10 02:40:57.947455 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:40:57.947427    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qtm94" event={"ID":"28f636a3-81c8-4492-a301-98f47fdc30d3","Type":"ContainerStarted","Data":"b2bdb80c08654b556d823bb7116a21d3cc960a0bfda10b8bb0ef098432135e65"}
Jul 10 02:41:02.965332 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.965297    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn7g5" event={"ID":"1018b000-e21e-4f77-806f-da744fa85e52","Type":"ContainerStarted","Data":"b1219dd007b42a22ccd68bd00625dc670f86b1ca00173db1d263d4e6ed92e66a"}
Jul 10 02:41:02.965757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.965348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dn7g5" event={"ID":"1018b000-e21e-4f77-806f-da744fa85e52","Type":"ContainerStarted","Data":"cbab464fe050b9a7a4b8b9d357df68435dbf7d807172d8bcfbb2162b36bfc986"}
Jul 10 02:41:02.965757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.965406    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:41:02.967091 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.967073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qtm94" event={"ID":"28f636a3-81c8-4492-a301-98f47fdc30d3","Type":"ContainerStarted","Data":"cb03ae612684c2c9d459d0412c388060c9fcaa9209cef85920146c10804efa8e"}
Jul 10 02:41:02.967245 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.967130    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qtm94" event={"ID":"28f636a3-81c8-4492-a301-98f47fdc30d3","Type":"ContainerStarted","Data":"556f59b1f1a4e0daf7d40c314a78f3922d90f7320e7665c08ce6cfc333b6086d"}
Jul 10 02:41:02.983920 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:02.983876    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dn7g5" podStartSLOduration=17.409469588 podStartE2EDuration="21.983860754s" totalImagesPullingTime="4.574391166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:02.983644944 +0000 UTC m=+56.796723034" watchObservedRunningTime="2026-07-10 02:41:02.983860754 +0000 UTC m=+56.796938845"
Jul 10 02:41:05.981209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:05.981171    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qtm94" event={"ID":"28f636a3-81c8-4492-a301-98f47fdc30d3","Type":"ContainerStarted","Data":"d67cc2bc9e12aeb55a2f4c0b3ca61d363a53ae4221d414652e81b121655314a3"}
Jul 10 02:41:05.997066 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:05.997014    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qtm94" podStartSLOduration=17.603813624 podStartE2EDuration="24.99696241s" totalImagesPullingTime="7.393148786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:05.99666613 +0000 UTC m=+59.809744220" watchObservedRunningTime="2026-07-10 02:41:05.99696241 +0000 UTC m=+59.810040496"
Jul 10 02:41:06.493996 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.493961    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"]
Jul 10 02:41:06.495898 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.495880    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.499156 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.499129    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"]
Jul 10 02:41:06.499699 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.499670    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 02:41:06.499844 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.499676    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tqzvz\""
Jul 10 02:41:06.500071 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.500044    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 02:41:06.500258 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.500238    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 10 02:41:06.500462 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.500446    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 02:41:06.501001 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.500976    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"]
Jul 10 02:41:06.502065 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.502049    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.503396 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.503378    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.504803 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.504783    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 10 02:41:06.505651 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.505631    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 02:41:06.506116 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.506078    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 02:41:06.506201 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.506151    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 02:41:06.506384 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.506366    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 02:41:06.509705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.509683    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"]
Jul 10 02:41:06.519303 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.519271    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"]
Jul 10 02:41:06.520422 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.520398    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"]
Jul 10 02:41:06.583426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.583426 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7snx\" (UniqueName: \"kubernetes.io/projected/d02c7d27-ce8c-4635-b112-fff6e8acb11f-kube-api-access-l7snx\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583510    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvptx\" (UniqueName: \"kubernetes.io/projected/ad78ccb4-777e-43aa-8051-d324db5b58c2-kube-api-access-kvptx\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583539    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/accfc9ae-adcd-4caa-914b-adb291a148d9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad78ccb4-777e-43aa-8051-d324db5b58c2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583593    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02c7d27-ce8c-4635-b112-fff6e8acb11f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.583653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583630    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.583927 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583674    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwzs\" (UniqueName: \"kubernetes.io/projected/accfc9ae-adcd-4caa-914b-adb291a148d9-kube-api-access-mbwzs\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.583927 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad78ccb4-777e-43aa-8051-d324db5b58c2-tmp\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.583927 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.583745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.684229 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684177    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/accfc9ae-adcd-4caa-914b-adb291a148d9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.684229 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad78ccb4-777e-43aa-8051-d324db5b58c2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.684484 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684267    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02c7d27-ce8c-4635-b112-fff6e8acb11f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.684484 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.684484 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwzs\" (UniqueName: \"kubernetes.io/projected/accfc9ae-adcd-4caa-914b-adb291a148d9-kube-api-access-mbwzs\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.684484 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad78ccb4-777e-43aa-8051-d324db5b58c2-tmp\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.684815 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.684784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad78ccb4-777e-43aa-8051-d324db5b58c2-tmp\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.685151 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685120    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02c7d27-ce8c-4635-b112-fff6e8acb11f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.685262 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685240    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.685326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.685377 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.685377 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7snx\" (UniqueName: \"kubernetes.io/projected/d02c7d27-ce8c-4635-b112-fff6e8acb11f-kube-api-access-l7snx\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.685483 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.685416    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvptx\" (UniqueName: \"kubernetes.io/projected/ad78ccb4-777e-43aa-8051-d324db5b58c2-kube-api-access-kvptx\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.687575 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.687518    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad78ccb4-777e-43aa-8051-d324db5b58c2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.687575 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.687531    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.687806 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.687787    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-ca\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.688174 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.688160    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.688229 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.688167    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d02c7d27-ce8c-4635-b112-fff6e8acb11f-hub\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.688267 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.688225    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/accfc9ae-adcd-4caa-914b-adb291a148d9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.707385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.707356    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7snx\" (UniqueName: \"kubernetes.io/projected/d02c7d27-ce8c-4635-b112-fff6e8acb11f-kube-api-access-l7snx\") pod \"cluster-proxy-proxy-agent-64fdc8986c-gdprz\" (UID: \"d02c7d27-ce8c-4635-b112-fff6e8acb11f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.707385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.707376    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvptx\" (UniqueName: \"kubernetes.io/projected/ad78ccb4-777e-43aa-8051-d324db5b58c2-kube-api-access-kvptx\") pod \"klusterlet-addon-workmgr-6954445bb6-497pl\" (UID: \"ad78ccb4-777e-43aa-8051-d324db5b58c2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.708830 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.708801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwzs\" (UniqueName: \"kubernetes.io/projected/accfc9ae-adcd-4caa-914b-adb291a148d9-kube-api-access-mbwzs\") pod \"managed-serviceaccount-addon-agent-584bb4bffd-pqvfm\" (UID: \"accfc9ae-adcd-4caa-914b-adb291a148d9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.818961 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.818872    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"
Jul 10 02:41:06.830515 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.830489    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:06.836781 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.836760    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"
Jul 10 02:41:06.972950 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.972879    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm"]
Jul 10 02:41:06.976995 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:06.976967    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaccfc9ae_adcd_4caa_914b_adb291a148d9.slice/crio-30c9b26845c997dec3bab0f153d35e3e810773b58cf26838260c2b19dd7c7d1d WatchSource:0}: Error finding container 30c9b26845c997dec3bab0f153d35e3e810773b58cf26838260c2b19dd7c7d1d: Status 404 returned error can't find the container with id 30c9b26845c997dec3bab0f153d35e3e810773b58cf26838260c2b19dd7c7d1d
Jul 10 02:41:06.985560 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:06.985532    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm" event={"ID":"accfc9ae-adcd-4caa-914b-adb291a148d9","Type":"ContainerStarted","Data":"30c9b26845c997dec3bab0f153d35e3e810773b58cf26838260c2b19dd7c7d1d"}
Jul 10 02:41:07.200001 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:07.199974    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"]
Jul 10 02:41:07.201213 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:07.201163    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz"]
Jul 10 02:41:07.202794 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:07.202766    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02c7d27_ce8c_4635_b112_fff6e8acb11f.slice/crio-996d3572546461e40e282cb9dda0786c8d23ddee729e81936123c513a6ae5876 WatchSource:0}: Error finding container 996d3572546461e40e282cb9dda0786c8d23ddee729e81936123c513a6ae5876: Status 404 returned error can't find the container with id 996d3572546461e40e282cb9dda0786c8d23ddee729e81936123c513a6ae5876
Jul 10 02:41:07.203352 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:07.203323    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad78ccb4_777e_43aa_8051_d324db5b58c2.slice/crio-59285373b9cff19070711a47a5a1299b2f39eb74d265bafd90f1624a8332d4be WatchSource:0}: Error finding container 59285373b9cff19070711a47a5a1299b2f39eb74d265bafd90f1624a8332d4be: Status 404 returned error can't find the container with id 59285373b9cff19070711a47a5a1299b2f39eb74d265bafd90f1624a8332d4be
Jul 10 02:41:07.889017 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:07.888992    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vhtt6"
Jul 10 02:41:07.990962 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:07.990904    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl" event={"ID":"ad78ccb4-777e-43aa-8051-d324db5b58c2","Type":"ContainerStarted","Data":"59285373b9cff19070711a47a5a1299b2f39eb74d265bafd90f1624a8332d4be"}
Jul 10 02:41:07.992385 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:07.992344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz" event={"ID":"d02c7d27-ce8c-4635-b112-fff6e8acb11f","Type":"ContainerStarted","Data":"996d3572546461e40e282cb9dda0786c8d23ddee729e81936123c513a6ae5876"}
Jul 10 02:41:12.442840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.442796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:41:12.443420 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.442883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:41:12.445555 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.445531    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 02:41:12.445670 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.445618    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 02:41:12.456446 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.456418    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6063745e-5b20-4c59-b4a0-3b7d34009b94-original-pull-secret\") pod \"global-pull-secret-syncer-mg6bl\" (UID: \"6063745e-5b20-4c59-b4a0-3b7d34009b94\") " pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:41:12.456588 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.456420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59d79d7-5041-45ff-90c2-9fb7ef380df9-metrics-certs\") pod \"network-metrics-daemon-5x97c\" (UID: \"d59d79d7-5041-45ff-90c2-9fb7ef380df9\") " pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:41:12.543705 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.543666    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:41:12.546315 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.546290    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 02:41:12.556804 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.556778    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 02:41:12.567304 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.567277    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f2j\" (UniqueName: \"kubernetes.io/projected/9a53e3f4-b9e5-46f5-a87a-8e73be826cfb-kube-api-access-p4f2j\") pod \"network-check-target-9rgvz\" (UID: \"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb\") " pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:41:12.595028 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.594987    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h9pgv\""
Jul 10 02:41:12.598197 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.598169    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jw76v\""
Jul 10 02:41:12.603186 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.603162    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:41:12.606500 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.606481    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5x97c"
Jul 10 02:41:12.695810 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.695720    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg6bl"
Jul 10 02:41:12.979227 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:12.978442    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dn7g5"
Jul 10 02:41:13.050563 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.045618    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:13.050563 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.046415    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl" podUID="ad78ccb4-777e-43aa-8051-d324db5b58c2" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.11:8000/readyz\": dial tcp 10.134.0.11:8000: connect: connection refused"
Jul 10 02:41:13.065969 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.065161    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl" podStartSLOduration=1.335453624 podStartE2EDuration="7.065137385s" totalImagesPullingTime="5.729683761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:13.064301402 +0000 UTC m=+66.877379494" watchObservedRunningTime="2026-07-10 02:41:13.065137385 +0000 UTC m=+66.878215479"
Jul 10 02:41:13.107044 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.106431    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5x97c"]
Jul 10 02:41:13.110466 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:13.110389    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d79d7_5041_45ff_90c2_9fb7ef380df9.slice/crio-341998da20ea5b2240778c332293153892c0dbbab5714324a778087f50b64d84 WatchSource:0}: Error finding container 341998da20ea5b2240778c332293153892c0dbbab5714324a778087f50b64d84: Status 404 returned error can't find the container with id 341998da20ea5b2240778c332293153892c0dbbab5714324a778087f50b64d84
Jul 10 02:41:13.126206 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.126155    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mg6bl"]
Jul 10 02:41:13.130536 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:13.130485    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6063745e_5b20_4c59_b4a0_3b7d34009b94.slice/crio-4191befba1f61f6aebca9e28e2adf5ed80cccd2c95a63303548a0c7a953d0106 WatchSource:0}: Error finding container 4191befba1f61f6aebca9e28e2adf5ed80cccd2c95a63303548a0c7a953d0106: Status 404 returned error can't find the container with id 4191befba1f61f6aebca9e28e2adf5ed80cccd2c95a63303548a0c7a953d0106
Jul 10 02:41:13.143846 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.143820    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rgvz"]
Jul 10 02:41:13.155464 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:13.155412    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53e3f4_b9e5_46f5_a87a_8e73be826cfb.slice/crio-559aaaebd6cac930d7463ef2946f6890422baa1436204a0fc59d4720acb3cb33 WatchSource:0}: Error finding container 559aaaebd6cac930d7463ef2946f6890422baa1436204a0fc59d4720acb3cb33: Status 404 returned error can't find the container with id 559aaaebd6cac930d7463ef2946f6890422baa1436204a0fc59d4720acb3cb33
Jul 10 02:41:13.759697 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.758448    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2b6n4"]
Jul 10 02:41:13.761008 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.760921    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.764826 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.764649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 02:41:13.766126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.765583    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dlwvz\""
Jul 10 02:41:13.766126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.765793    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 02:41:13.766126 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.766012    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 02:41:13.770389 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.769538    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 02:41:13.770389 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.769740    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 02:41:13.770389 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.769919    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.855996    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-wtmp\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856051    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856084    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-textfile\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856126    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-metrics-client-ca\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856154    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-root\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-sys\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856259    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.856405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.856290    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54r82\" (UniqueName: \"kubernetes.io/projected/b2077661-f3aa-45b4-951f-90e85a4300b6-kube-api-access-54r82\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.956838    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-root\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.956889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.956943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-sys\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957022    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-root\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957091    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-sys\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957144    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957184    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54r82\" (UniqueName: \"kubernetes.io/projected/b2077661-f3aa-45b4-951f-90e85a4300b6-kube-api-access-54r82\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957238    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-wtmp\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957267    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-textfile\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.957441 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-metrics-client-ca\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.957965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-metrics-client-ca\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.958009    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-wtmp\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:41:13.958210    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.958268    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-textfile\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:41:13.958279    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls podName:b2077661-f3aa-45b4-951f-90e85a4300b6 nodeName:}" failed. No retries permitted until 2026-07-10 02:41:14.45825824 +0000 UTC m=+68.271336316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls") pod "node-exporter-2b6n4" (UID: "b2077661-f3aa-45b4-951f-90e85a4300b6") : secret "node-exporter-tls" not found
Jul 10 02:41:13.958648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.958463    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.961780 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.961759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:13.980685 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:13.980596    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54r82\" (UniqueName: \"kubernetes.io/projected/b2077661-f3aa-45b4-951f-90e85a4300b6-kube-api-access-54r82\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:14.058511 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.058374    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5x97c" event={"ID":"d59d79d7-5041-45ff-90c2-9fb7ef380df9","Type":"ContainerStarted","Data":"341998da20ea5b2240778c332293153892c0dbbab5714324a778087f50b64d84"}
Jul 10 02:41:14.062064 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.061971    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm" event={"ID":"accfc9ae-adcd-4caa-914b-adb291a148d9","Type":"ContainerStarted","Data":"e4dc8f49b4f03c5751ef27d19846f05af742d681d42830492a7f8c6c4779053a"}
Jul 10 02:41:14.067341 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.066202    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl" event={"ID":"ad78ccb4-777e-43aa-8051-d324db5b58c2","Type":"ContainerStarted","Data":"51f540f3f6c1bb36e4bdc030f82efa1b91c24ced577e203ebde8f8e2a68ff9b4"}
Jul 10 02:41:14.068545 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.068491    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6954445bb6-497pl"
Jul 10 02:41:14.071315 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.071285    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rgvz" event={"ID":"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb","Type":"ContainerStarted","Data":"559aaaebd6cac930d7463ef2946f6890422baa1436204a0fc59d4720acb3cb33"}
Jul 10 02:41:14.075820 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.074911    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz" event={"ID":"d02c7d27-ce8c-4635-b112-fff6e8acb11f","Type":"ContainerStarted","Data":"6f2927b2943d0603fb9e59c4d7b4b4589b190d707f18375fbb36088656ae8b1f"}
Jul 10 02:41:14.075820 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.074961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz" event={"ID":"d02c7d27-ce8c-4635-b112-fff6e8acb11f","Type":"ContainerStarted","Data":"e5221f199cd31e1c36f39dbc1fc823c84b2cf56571bd1fb0470f5b3f85483b3b"}
Jul 10 02:41:14.075820 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.074980    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz" event={"ID":"d02c7d27-ce8c-4635-b112-fff6e8acb11f","Type":"ContainerStarted","Data":"bb9efec837c78e86fa9e63a93c9d15fd4ceeb8af7b939f83b4ee407d56122e41"}
Jul 10 02:41:14.077946 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.077869    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mg6bl" event={"ID":"6063745e-5b20-4c59-b4a0-3b7d34009b94","Type":"ContainerStarted","Data":"4191befba1f61f6aebca9e28e2adf5ed80cccd2c95a63303548a0c7a953d0106"}
Jul 10 02:41:14.101111 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.100887    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584bb4bffd-pqvfm" podStartSLOduration=2.165538071 podStartE2EDuration="8.10086513s" totalImagesPullingTime="5.935327059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:14.080687008 +0000 UTC m=+67.893765102" watchObservedRunningTime="2026-07-10 02:41:14.10086513 +0000 UTC m=+67.913943223"
Jul 10 02:41:14.106809 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.106369    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64fdc8986c-gdprz" podStartSLOduration=2.380692552 podStartE2EDuration="8.106288026s" totalImagesPullingTime="5.725595474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:14.105027639 +0000 UTC m=+67.918105734" watchObservedRunningTime="2026-07-10 02:41:14.106288026 +0000 UTC m=+67.919366119"
Jul 10 02:41:14.461924 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.461425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:14.468620 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.468547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2077661-f3aa-45b4-951f-90e85a4300b6-node-exporter-tls\") pod \"node-exporter-2b6n4\" (UID: \"b2077661-f3aa-45b4-951f-90e85a4300b6\") " pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:14.680636 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:14.680600    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2b6n4"
Jul 10 02:41:14.707195 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:41:14.707141    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2077661_f3aa_45b4_951f_90e85a4300b6.slice/crio-289a713195f209f0368230257c1bd1c53340458eb7a969ba40149766eb85c31c WatchSource:0}: Error finding container 289a713195f209f0368230257c1bd1c53340458eb7a969ba40149766eb85c31c: Status 404 returned error can't find the container with id 289a713195f209f0368230257c1bd1c53340458eb7a969ba40149766eb85c31c
Jul 10 02:41:15.085116 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:15.085052    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5x97c" event={"ID":"d59d79d7-5041-45ff-90c2-9fb7ef380df9","Type":"ContainerStarted","Data":"52a7582d9bdc413f705ada7d3f4302d16458a5022d13ab82c6322045c193231a"}
Jul 10 02:41:15.086609 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:15.086581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b6n4" event={"ID":"b2077661-f3aa-45b4-951f-90e85a4300b6","Type":"ContainerStarted","Data":"289a713195f209f0368230257c1bd1c53340458eb7a969ba40149766eb85c31c"}
Jul 10 02:41:16.092745 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:16.092712    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5x97c" event={"ID":"d59d79d7-5041-45ff-90c2-9fb7ef380df9","Type":"ContainerStarted","Data":"038f5ca95e3d2b51ec9a0e5f29f44fbaf2253ef12d77e08f4249cca1b90263e0"}
Jul 10 02:41:16.109320 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:16.109270    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5x97c" podStartSLOduration=67.313797279 podStartE2EDuration="1m9.109252585s" totalImagesPullingTime="1.795455306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:16.10734324 +0000 UTC m=+69.920421330" watchObservedRunningTime="2026-07-10 02:41:16.109252585 +0000 UTC m=+69.922330675"
Jul 10 02:41:19.105329 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.105233    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rgvz" event={"ID":"9a53e3f4-b9e5-46f5-a87a-8e73be826cfb","Type":"ContainerStarted","Data":"139f1ea997c69f8d137ed227aa6f36093b623707a425de804a902b62978ebe9e"}
Jul 10 02:41:19.105329 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.105319    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:41:19.106881 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.106853    2551 generic.go:356] "Generic (PLEG): container finished" podID="b2077661-f3aa-45b4-951f-90e85a4300b6" containerID="28e371409cae250cff786558f7cb06bcb1c0d11be6c34df10080ff5868bd7f72" exitCode=0
Jul 10 02:41:19.106996 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.106945    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b6n4" event={"ID":"b2077661-f3aa-45b4-951f-90e85a4300b6","Type":"ContainerDied","Data":"28e371409cae250cff786558f7cb06bcb1c0d11be6c34df10080ff5868bd7f72"}
Jul 10 02:41:19.108348 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.108326    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mg6bl" event={"ID":"6063745e-5b20-4c59-b4a0-3b7d34009b94","Type":"ContainerStarted","Data":"c9fc1678fc93eff8a1adffff719489564089eeba6bd4d6ee2d0687dd78eb2f03"}
Jul 10 02:41:19.120036 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.119979    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9rgvz" podStartSLOduration=66.434052506 podStartE2EDuration="1m12.119960188s" totalImagesPullingTime="5.685907682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:19.118778526 +0000 UTC m=+72.931856618" watchObservedRunningTime="2026-07-10 02:41:19.119960188 +0000 UTC m=+72.933038280"
Jul 10 02:41:19.151439 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:19.151207    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mg6bl" podStartSLOduration=66.437774942 podStartE2EDuration="1m12.151188773s" totalImagesPullingTime="5.713413831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:19.150818943 +0000 UTC m=+72.963897035" watchObservedRunningTime="2026-07-10 02:41:19.151188773 +0000 UTC m=+72.964266866"
Jul 10 02:41:20.114760 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:20.114723    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b6n4" event={"ID":"b2077661-f3aa-45b4-951f-90e85a4300b6","Type":"ContainerStarted","Data":"db236a0793d6f886159111379a9d084a2d0bc7dcfd840948e3d62f2b80c0026f"}
Jul 10 02:41:20.115153 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:20.114783    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b6n4" event={"ID":"b2077661-f3aa-45b4-951f-90e85a4300b6","Type":"ContainerStarted","Data":"652cab89c98515398334f3868d7088fd21de1a6a729937fca98d6d021b194f9a"}
Jul 10 02:41:20.137653 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:20.137602    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2b6n4" podStartSLOduration=2.950229813 podStartE2EDuration="7.137581521s" totalImagesPullingTime="4.119489687s" totalInitContainerRuntime="67.862021ms" isStatefulPod=true podCreationTimestamp="2026-07-10 02:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:20.135634857 +0000 UTC m=+73.948712973" watchObservedRunningTime="2026-07-10 02:41:20.137581521 +0000 UTC m=+73.950659611"
Jul 10 02:41:50.118236 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:41:50.118204    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9rgvz"
Jul 10 02:45:06.513220 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:06.513186    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:45:06.516465 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:06.516442    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:45:06.517024 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:06.517005    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 02:45:49.946786 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.946751    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"]
Jul 10 02:45:49.951282 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.949785    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.953360 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.953324    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" podUID="1c04809c-a030-4044-8cb2-db346ed54d73" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.953488 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.953384    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-q69t7\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q69t7\"" type="*v1.Secret"
Jul 10 02:45:49.953769 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.953749    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Jul 10 02:45:49.954084 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.954066    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 10 02:45:49.955296 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.955276    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 10 02:45:49.955576 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:49.955556    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Jul 10 02:45:49.987061 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.987025    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.987061 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.987062    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-tmp\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.987299 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.987105    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.987299 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.987156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmjf\" (UniqueName: \"kubernetes.io/projected/1c04809c-a030-4044-8cb2-db346ed54d73-kube-api-access-gsmjf\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:49.987299 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:49.987223    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-manifests\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.006269 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.006230    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"]
Jul 10 02:45:50.088072 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088038    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-manifests\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.088271 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088083    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.088271 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088211    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-tmp\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.088271 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.088405 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmjf\" (UniqueName: \"kubernetes.io/projected/1c04809c-a030-4044-8cb2-db346ed54d73-kube-api-access-gsmjf\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.088553 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.088530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-manifests\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.090418 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.090400    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c04809c-a030-4044-8cb2-db346ed54d73-tmp\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:50.847742 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:50.847710    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 10 02:45:51.013235 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.013206    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 10 02:45:51.019517 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.019482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmjf\" (UniqueName: \"kubernetes.io/projected/1c04809c-a030-4044-8cb2-db346ed54d73-kube-api-access-gsmjf\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.088841 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:51.088803    2551 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 10 02:45:51.089044 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:51.088812    2551 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 10 02:45:51.089044 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:51.088913    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert podName:1c04809c-a030-4044-8cb2-db346ed54d73 nodeName:}" failed. No retries permitted until 2026-07-10 02:45:51.588892145 +0000 UTC m=+345.401970214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert") pod "opendatahub-operator-controller-manager-68647b69dd-nsr52" (UID: "1c04809c-a030-4044-8cb2-db346ed54d73") : failed to sync secret cache: timed out waiting for the condition
Jul 10 02:45:51.089044 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:45:51.088952    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert podName:1c04809c-a030-4044-8cb2-db346ed54d73 nodeName:}" failed. No retries permitted until 2026-07-10 02:45:51.58893916 +0000 UTC m=+345.402017230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert") pod "opendatahub-operator-controller-manager-68647b69dd-nsr52" (UID: "1c04809c-a030-4044-8cb2-db346ed54d73") : failed to sync secret cache: timed out waiting for the condition
Jul 10 02:45:51.125818 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.125725    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q69t7\""
Jul 10 02:45:51.316816 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.316782    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 10 02:45:51.513813 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.513782    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 10 02:45:51.600388 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.600350    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.600545 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.600405    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.602809 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.602790    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.602859 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.602820    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c04809c-a030-4044-8cb2-db346ed54d73-webhook-cert\") pod \"opendatahub-operator-controller-manager-68647b69dd-nsr52\" (UID: \"1c04809c-a030-4044-8cb2-db346ed54d73\") " pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.760914 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.760877    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:51.897822 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.897798    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"]
Jul 10 02:45:51.900240 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:45:51.900211    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c04809c_a030_4044_8cb2_db346ed54d73.slice/crio-35a55549d2bed3576154e5dcabcd483221b8a27a48397e926756780e66508197 WatchSource:0}: Error finding container 35a55549d2bed3576154e5dcabcd483221b8a27a48397e926756780e66508197: Status 404 returned error can't find the container with id 35a55549d2bed3576154e5dcabcd483221b8a27a48397e926756780e66508197
Jul 10 02:45:51.901937 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.901919    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 02:45:51.999198 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:51.999170    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52" event={"ID":"1c04809c-a030-4044-8cb2-db346ed54d73","Type":"ContainerStarted","Data":"35a55549d2bed3576154e5dcabcd483221b8a27a48397e926756780e66508197"}
Jul 10 02:45:55.012295 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:55.012257    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52" event={"ID":"1c04809c-a030-4044-8cb2-db346ed54d73","Type":"ContainerStarted","Data":"5e21d72cf646f720a33a6936d4c97e2bd850135c89f9d4fa12cd0268aed67907"}
Jul 10 02:45:56.017011 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:56.016915    2551 generic.go:356] "Generic (PLEG): container finished" podID="1c04809c-a030-4044-8cb2-db346ed54d73" containerID="5e21d72cf646f720a33a6936d4c97e2bd850135c89f9d4fa12cd0268aed67907" exitCode=0
Jul 10 02:45:56.017011 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:56.016966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52" event={"ID":"1c04809c-a030-4044-8cb2-db346ed54d73","Type":"ContainerDied","Data":"5e21d72cf646f720a33a6936d4c97e2bd850135c89f9d4fa12cd0268aed67907"}
Jul 10 02:45:57.021867 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:57.021833    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52" event={"ID":"1c04809c-a030-4044-8cb2-db346ed54d73","Type":"ContainerStarted","Data":"7db8b49533be929106130aaa8652e5731bd04d926db83ac5b865cb2f23eaa7a3"}
Jul 10 02:45:57.022267 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:57.022056    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:45:57.050857 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:45:57.050798    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52" podStartSLOduration=3.973886027 podStartE2EDuration="8.050777762s" totalImagesPullingTime="2.991795738s" totalInitContainerRuntime="1.085095997s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:45:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:45:57.048498322 +0000 UTC m=+350.861576412" watchObservedRunningTime="2026-07-10 02:45:57.050777762 +0000 UTC m=+350.863855854"
Jul 10 02:46:08.027389 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:08.027308    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68647b69dd-nsr52"
Jul 10 02:46:10.604680 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.604647    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"]
Jul 10 02:46:10.614297 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.614268    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.618292 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.618269    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 10 02:46:10.618531 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.618517    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vjqd4\""
Jul 10 02:46:10.620042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.620020    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 10 02:46:10.621755 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.621225    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 10 02:46:10.625151 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.625132    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 10 02:46:10.630966 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.630937    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"]
Jul 10 02:46:10.745228 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.745187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c083487-9367-4363-a124-a2328ce7de59-tmp\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.745390 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.745261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c083487-9367-4363-a124-a2328ce7de59-tls-certs\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.745390 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.745324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2lw\" (UniqueName: \"kubernetes.io/projected/0c083487-9367-4363-a124-a2328ce7de59-kube-api-access-8c2lw\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.816353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.816312    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r4m57"]
Jul 10 02:46:10.818264 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.818247    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:10.822114 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.822072    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 10 02:46:10.822253 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.822184    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9t9m7\""
Jul 10 02:46:10.845985 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.845950    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2lw\" (UniqueName: \"kubernetes.io/projected/0c083487-9367-4363-a124-a2328ce7de59-kube-api-access-8c2lw\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.846208 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.846002    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c083487-9367-4363-a124-a2328ce7de59-tmp\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.846208 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.846078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c083487-9367-4363-a124-a2328ce7de59-tls-certs\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.847439 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.847416    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r4m57"]
Jul 10 02:46:10.848536 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.848515    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c083487-9367-4363-a124-a2328ce7de59-tmp\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.848794 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.848777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c083487-9367-4363-a124-a2328ce7de59-tls-certs\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.865788 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.865712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2lw\" (UniqueName: \"kubernetes.io/projected/0c083487-9367-4363-a124-a2328ce7de59-kube-api-access-8c2lw\") pod \"kube-auth-proxy-5d56d99d47-5trfj\" (UID: \"0c083487-9367-4363-a124-a2328ce7de59\") " pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.925145 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.925084    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"
Jul 10 02:46:10.947200 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.947169    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:10.947367 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:10.947235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchp5\" (UniqueName: \"kubernetes.io/projected/63322fa8-8b22-43c8-9938-f94c636f0d4e-kube-api-access-rchp5\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.048641 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.048608    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchp5\" (UniqueName: \"kubernetes.io/projected/63322fa8-8b22-43c8-9938-f94c636f0d4e-kube-api-access-rchp5\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.048819 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.048655    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.048819 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:46:11.048763    2551 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 10 02:46:11.048905 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:46:11.048822    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert podName:63322fa8-8b22-43c8-9938-f94c636f0d4e nodeName:}" failed. No retries permitted until 2026-07-10 02:46:11.54880521 +0000 UTC m=+365.361883304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert") pod "odh-model-controller-7998c67dc4-r4m57" (UID: "63322fa8-8b22-43c8-9938-f94c636f0d4e") : secret "odh-model-controller-webhook-cert" not found
Jul 10 02:46:11.062294 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.062262    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj"]
Jul 10 02:46:11.064240 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.064221    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchp5\" (UniqueName: \"kubernetes.io/projected/63322fa8-8b22-43c8-9938-f94c636f0d4e-kube-api-access-rchp5\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.066389 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:46:11.066363    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c083487_9367_4363_a124_a2328ce7de59.slice/crio-3abb26b8f5eee58da6cde7ab8a167e4d5fb88baab177493e06f04f5de169c470 WatchSource:0}: Error finding container 3abb26b8f5eee58da6cde7ab8a167e4d5fb88baab177493e06f04f5de169c470: Status 404 returned error can't find the container with id 3abb26b8f5eee58da6cde7ab8a167e4d5fb88baab177493e06f04f5de169c470
Jul 10 02:46:11.072129 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.072087    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj" event={"ID":"0c083487-9367-4363-a124-a2328ce7de59","Type":"ContainerStarted","Data":"3abb26b8f5eee58da6cde7ab8a167e4d5fb88baab177493e06f04f5de169c470"}
Jul 10 02:46:11.553364 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.553322    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.555994 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.555956    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63322fa8-8b22-43c8-9938-f94c636f0d4e-cert\") pod \"odh-model-controller-7998c67dc4-r4m57\" (UID: \"63322fa8-8b22-43c8-9938-f94c636f0d4e\") " pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.727983 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.727945    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:11.798458 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.798393    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-h5t8d"]
Jul 10 02:46:11.801267 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.801241    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:11.807874 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.807575    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jhtrl\""
Jul 10 02:46:11.807874 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.807612    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 10 02:46:11.828760 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.828621    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-h5t8d"]
Jul 10 02:46:11.898115 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.898058    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r4m57"]
Jul 10 02:46:11.901356 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:46:11.901315    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63322fa8_8b22_43c8_9938_f94c636f0d4e.slice/crio-62e3a3aa206adf904aaa9a4aa5e9b39c0ea641ba2305d3c7d32216b5f7bfa0c6 WatchSource:0}: Error finding container 62e3a3aa206adf904aaa9a4aa5e9b39c0ea641ba2305d3c7d32216b5f7bfa0c6: Status 404 returned error can't find the container with id 62e3a3aa206adf904aaa9a4aa5e9b39c0ea641ba2305d3c7d32216b5f7bfa0c6
Jul 10 02:46:11.957113 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.957064    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7vh\" (UniqueName: \"kubernetes.io/projected/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-kube-api-access-5h7vh\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:11.957280 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:11.957163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-tls-certs\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.058515 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.058421    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7vh\" (UniqueName: \"kubernetes.io/projected/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-kube-api-access-5h7vh\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.058515 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.058471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-tls-certs\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.060941 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.060920    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-tls-certs\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.069404 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.069378    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7vh\" (UniqueName: \"kubernetes.io/projected/8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1-kube-api-access-5h7vh\") pod \"model-serving-api-54d89bd5fc-h5t8d\" (UID: \"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1\") " pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.077816 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.077786    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57" event={"ID":"63322fa8-8b22-43c8-9938-f94c636f0d4e","Type":"ContainerStarted","Data":"62e3a3aa206adf904aaa9a4aa5e9b39c0ea641ba2305d3c7d32216b5f7bfa0c6"}
Jul 10 02:46:12.115173 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.115138    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:12.244178 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.244146    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:12.247334 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.247313    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.249951 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.249925    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 10 02:46:12.250090 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.250012    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 10 02:46:12.250909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.250891    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 10 02:46:12.250909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.250901    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6mzt5\""
Jul 10 02:46:12.251065 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.250943    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 10 02:46:12.264664 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.264630    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:12.289682 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.288120    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-h5t8d"]
Jul 10 02:46:12.291555 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:46:12.291522    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed35d12_3f4d_4c2f_8acd_89dd622ce9e1.slice/crio-7059b9e213de95e48725d155d5cee517983458e84a81ca2f7bba0f066d577a69 WatchSource:0}: Error finding container 7059b9e213de95e48725d155d5cee517983458e84a81ca2f7bba0f066d577a69: Status 404 returned error can't find the container with id 7059b9e213de95e48725d155d5cee517983458e84a81ca2f7bba0f066d577a69
Jul 10 02:46:12.361083 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.360985    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361083 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361083 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361082    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp44k\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361387 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361194    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361387 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361289    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361387 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361322    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.361387 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.361371    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.462787 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.462787 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462784    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp44k\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462833    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462857    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462873    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463042 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.462903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.463617 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.463589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.465244 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.465220    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.465391 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.465374    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.465576 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.465558    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.465644 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.465617    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.478933 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.478908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.484207 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.484184    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp44k\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k\") pod \"istiod-openshift-gateway-7574459656-4cjhh\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.557270 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.557236    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:12.732794 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:12.732709    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:12.737848 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:46:12.737816    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f225b7_ac13_419f_9d2f_bfda9c23a45c.slice/crio-5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026 WatchSource:0}: Error finding container 5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026: Status 404 returned error can't find the container with id 5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026
Jul 10 02:46:13.098074 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:13.097940    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" event={"ID":"50f225b7-ac13-419f-9d2f-bfda9c23a45c","Type":"ContainerStarted","Data":"5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026"}
Jul 10 02:46:13.102092 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:13.102014    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d" event={"ID":"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1","Type":"ContainerStarted","Data":"7059b9e213de95e48725d155d5cee517983458e84a81ca2f7bba0f066d577a69"}
Jul 10 02:46:16.816479 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:16.816432    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 10 02:46:16.816861 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:16.816513    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 10 02:46:17.119528 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.119425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57" event={"ID":"63322fa8-8b22-43c8-9938-f94c636f0d4e","Type":"ContainerStarted","Data":"aa5dac1077b3c9d988118e2dfda5d5ca85a894f6ff3ca6244fb3a5f9e2803522"}
Jul 10 02:46:17.119805 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.119555    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:17.121157 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.121088    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" event={"ID":"50f225b7-ac13-419f-9d2f-bfda9c23a45c","Type":"ContainerStarted","Data":"9a872f3eb467be081b77b0527a2a0943e670f33cdc054bc9c009a1da4ae231d5"}
Jul 10 02:46:17.121315 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.121285    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:17.123262 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.123235    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d" event={"ID":"8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1","Type":"ContainerStarted","Data":"67eb8684b9de3c85b1b9bd3d311262ffba0417bf6ac0dca2ecf213bb00be92c1"}
Jul 10 02:46:17.123457 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.123441    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:17.123578 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.123458    2551 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4cjhh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 10 02:46:17.123652 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.123616    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 02:46:17.124863 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.124842    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj" event={"ID":"0c083487-9367-4363-a124-a2328ce7de59","Type":"ContainerStarted","Data":"c3a021d0c7702b6955f237c633a2fe10a17c92f3c3f7b29350cfdae3d0151722"}
Jul 10 02:46:17.138599 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.138537    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57" podStartSLOduration=2.277241501 podStartE2EDuration="7.138517087s" totalImagesPullingTime="4.861275586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:46:17.138426753 +0000 UTC m=+370.951504845" watchObservedRunningTime="2026-07-10 02:46:17.138517087 +0000 UTC m=+370.951595212"
Jul 10 02:46:17.160407 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.160348    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d" podStartSLOduration=1.7001665240000001 podStartE2EDuration="6.160327765s" totalImagesPullingTime="4.460161241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:46:17.16023285 +0000 UTC m=+370.973310943" watchObservedRunningTime="2026-07-10 02:46:17.160327765 +0000 UTC m=+370.973405858"
Jul 10 02:46:17.179264 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.179202    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d56d99d47-5trfj" podStartSLOduration=1.49389492 podStartE2EDuration="7.179181139s" totalImagesPullingTime="5.685286219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:46:17.178182137 +0000 UTC m=+370.991260229" watchObservedRunningTime="2026-07-10 02:46:17.179181139 +0000 UTC m=+370.992259231"
Jul 10 02:46:17.202963 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:17.202903    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" podStartSLOduration=1.127617116 podStartE2EDuration="5.202882665s" totalImagesPullingTime="4.075265549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:46:17.202148495 +0000 UTC m=+371.015226585" watchObservedRunningTime="2026-07-10 02:46:17.202882665 +0000 UTC m=+371.015960757"
Jul 10 02:46:18.129166 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:18.129134    2551 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4cjhh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 10 02:46:18.129546 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:18.129186    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 02:46:19.825890 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.825853    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"]
Jul 10 02:46:19.828864 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.828840    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:19.832580 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.832554    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 10 02:46:19.832911 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.832897    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 10 02:46:19.833551 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.833536    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 10 02:46:19.833665 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.833570    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 10 02:46:19.833665 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.833611    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x7tbt\""
Jul 10 02:46:19.833665 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.833635    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 10 02:46:19.925353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.925315    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"]
Jul 10 02:46:19.932021 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.931986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-metrics-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:19.932196 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.932034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b4d35e1-694c-4e3b-b166-00502d6b2704-manager-config\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:19.932196 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.932162    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmch9\" (UniqueName: \"kubernetes.io/projected/2b4d35e1-694c-4e3b-b166-00502d6b2704-kube-api-access-xmch9\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:19.932276 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:19.932228    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.033008 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.032966    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.033008 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.033014    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-metrics-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.033276 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.033038    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b4d35e1-694c-4e3b-b166-00502d6b2704-manager-config\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.033276 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.033250    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmch9\" (UniqueName: \"kubernetes.io/projected/2b4d35e1-694c-4e3b-b166-00502d6b2704-kube-api-access-xmch9\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.033847 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.033826    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b4d35e1-694c-4e3b-b166-00502d6b2704-manager-config\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.035473 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.035453    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.035666 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.035645    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4d35e1-694c-4e3b-b166-00502d6b2704-metrics-cert\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.063762 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.063724    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmch9\" (UniqueName: \"kubernetes.io/projected/2b4d35e1-694c-4e3b-b166-00502d6b2704-kube-api-access-xmch9\") pod \"lws-controller-manager-b767bdcf8-7rngr\" (UID: \"2b4d35e1-694c-4e3b-b166-00502d6b2704\") " pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.137528 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.137504    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:20.279277 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:20.279253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"]
Jul 10 02:46:20.281617 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:46:20.281595    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4d35e1_694c_4e3b_b166_00502d6b2704.slice/crio-ca4f114bfc7ffcc3f3c76e58dc6907dcd30f431b17d1088a047f2a569068c603 WatchSource:0}: Error finding container ca4f114bfc7ffcc3f3c76e58dc6907dcd30f431b17d1088a047f2a569068c603: Status 404 returned error can't find the container with id ca4f114bfc7ffcc3f3c76e58dc6907dcd30f431b17d1088a047f2a569068c603
Jul 10 02:46:21.130140 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:21.130113    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:21.140310 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:21.140270    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr" event={"ID":"2b4d35e1-694c-4e3b-b166-00502d6b2704","Type":"ContainerStarted","Data":"ca4f114bfc7ffcc3f3c76e58dc6907dcd30f431b17d1088a047f2a569068c603"}
Jul 10 02:46:23.150265 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:23.150236    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr" event={"ID":"2b4d35e1-694c-4e3b-b166-00502d6b2704","Type":"ContainerStarted","Data":"e48be1f1e85ad20ecf9bfee8527ebf2bcc2a579fda9df6d4c0257351309de921"}
Jul 10 02:46:23.150720 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:23.150370    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:46:23.180031 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:23.179956    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr" podStartSLOduration=1.614218921 podStartE2EDuration="4.179936232s" totalImagesPullingTime="2.565717311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:46:23.179282199 +0000 UTC m=+376.992360291" watchObservedRunningTime="2026-07-10 02:46:23.179936232 +0000 UTC m=+376.993014324"
Jul 10 02:46:23.917591 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:23.917556    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:23.917801 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:23.917777    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery" containerID="cri-o://9a872f3eb467be081b77b0527a2a0943e670f33cdc054bc9c009a1da4ae231d5" gracePeriod=30
Jul 10 02:46:24.155487 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.155458    2551 generic.go:356] "Generic (PLEG): container finished" podID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerID="9a872f3eb467be081b77b0527a2a0943e670f33cdc054bc9c009a1da4ae231d5" exitCode=0
Jul 10 02:46:24.155838 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.155532    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" event={"ID":"50f225b7-ac13-419f-9d2f-bfda9c23a45c","Type":"ContainerDied","Data":"9a872f3eb467be081b77b0527a2a0943e670f33cdc054bc9c009a1da4ae231d5"}
Jul 10 02:46:24.155838 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.155581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" event={"ID":"50f225b7-ac13-419f-9d2f-bfda9c23a45c","Type":"ContainerDied","Data":"5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026"}
Jul 10 02:46:24.155838 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.155595    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5176f7e5a2aaef4a5c4a1554873886d2cbf8bf0df801dfb39ce2f7baaa639026"
Jul 10 02:46:24.159850 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.159831    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:24.174768 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174710    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174768 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174748    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp44k\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174939 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174939 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174801    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174939 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174818    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174939 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174834    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.174939 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.174861    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token\") pod \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\" (UID: \"50f225b7-ac13-419f-9d2f-bfda9c23a45c\") "
Jul 10 02:46:24.175346 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.175265    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 02:46:24.177258 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177230    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 02:46:24.177258 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177248    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts" (OuterVolumeSpecName: "cacerts") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 02:46:24.177436 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177408    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k" (OuterVolumeSpecName: "kube-api-access-tp44k") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "kube-api-access-tp44k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:46:24.177475 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177444    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs" (OuterVolumeSpecName: "local-certs") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 02:46:24.177605 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177584    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 02:46:24.177648 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.177640    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token" (OuterVolumeSpecName: "istio-token") pod "50f225b7-ac13-419f-9d2f-bfda9c23a45c" (UID: "50f225b7-ac13-419f-9d2f-bfda9c23a45c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:46:24.276278 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276243    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-dns-cert\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276278 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276270    2551 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-kubeconfig\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276278 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276280    2551 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f225b7-ac13-419f-9d2f-bfda9c23a45c-local-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276509 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276291    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-csr-ca-configmap\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276509 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276305    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-istio-token\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276509 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276313    2551 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f225b7-ac13-419f-9d2f-bfda9c23a45c-cacerts\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:24.276509 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:24.276321    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp44k\" (UniqueName: \"kubernetes.io/projected/50f225b7-ac13-419f-9d2f-bfda9c23a45c-kube-api-access-tp44k\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:46:25.159311 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:25.159283    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"
Jul 10 02:46:25.177650 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:25.177608    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:25.181974 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:25.181947    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh"]
Jul 10 02:46:26.687143 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:26.687092    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" path="/var/lib/kubelet/pods/50f225b7-ac13-419f-9d2f-bfda9c23a45c/volumes"
Jul 10 02:46:28.131186 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:28.131157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r4m57"
Jul 10 02:46:28.133292 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:28.133276    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-h5t8d"
Jul 10 02:46:29.129564 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:29.129523    2551 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4cjhh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.17:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 10 02:46:29.129746 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:29.129589    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4cjhh" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.17:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 10 02:46:34.158492 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:46:34.158461    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b767bdcf8-7rngr"
Jul 10 02:48:00.165762 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.165726    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"]
Jul 10 02:48:00.166300 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.166160    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery"
Jul 10 02:48:00.166300 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.166178    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery"
Jul 10 02:48:00.166300 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.166274    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f225b7-ac13-419f-9d2f-bfda9c23a45c" containerName="discovery"
Jul 10 02:48:00.168977 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.168957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.171744 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.171721    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 10 02:48:00.171865 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.171748    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 10 02:48:00.171865 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.171774    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 10 02:48:00.172497 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.172484    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 10 02:48:00.172547 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.172499    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xwllg\""
Jul 10 02:48:00.177553 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.177531    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"]
Jul 10 02:48:00.183775 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.183751    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27cf62e1-8629-41e0-a2ba-6909d779857e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.183943 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.183786    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.183943 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.183822    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbxs\" (UniqueName: \"kubernetes.io/projected/27cf62e1-8629-41e0-a2ba-6909d779857e-kube-api-access-nvbxs\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.284571 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.284533    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbxs\" (UniqueName: \"kubernetes.io/projected/27cf62e1-8629-41e0-a2ba-6909d779857e-kube-api-access-nvbxs\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.284758 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.284598    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27cf62e1-8629-41e0-a2ba-6909d779857e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.284758 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.284625    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.284758 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:48:00.284712    2551 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 10 02:48:00.284883 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:48:00.284765    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert podName:27cf62e1-8629-41e0-a2ba-6909d779857e nodeName:}" failed. No retries permitted until 2026-07-10 02:48:00.784749259 +0000 UTC m=+474.597827327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rz4wk" (UID: "27cf62e1-8629-41e0-a2ba-6909d779857e") : secret "plugin-serving-cert" not found
Jul 10 02:48:00.285464 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.285442    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27cf62e1-8629-41e0-a2ba-6909d779857e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.293598 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.293570    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbxs\" (UniqueName: \"kubernetes.io/projected/27cf62e1-8629-41e0-a2ba-6909d779857e-kube-api-access-nvbxs\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.791784 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.791731    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:00.794369 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:00.794330    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27cf62e1-8629-41e0-a2ba-6909d779857e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rz4wk\" (UID: \"27cf62e1-8629-41e0-a2ba-6909d779857e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:01.078958 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:01.078868    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"
Jul 10 02:48:01.210794 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:01.210631    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk"]
Jul 10 02:48:01.213360 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:48:01.213324    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27cf62e1_8629_41e0_a2ba_6909d779857e.slice/crio-7006532150d9976608f84a98d7cf7c9e11b1a2b673cd23fc10dfc635ff2ad897 WatchSource:0}: Error finding container 7006532150d9976608f84a98d7cf7c9e11b1a2b673cd23fc10dfc635ff2ad897: Status 404 returned error can't find the container with id 7006532150d9976608f84a98d7cf7c9e11b1a2b673cd23fc10dfc635ff2ad897
Jul 10 02:48:01.512001 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:01.511964    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk" event={"ID":"27cf62e1-8629-41e0-a2ba-6909d779857e","Type":"ContainerStarted","Data":"7006532150d9976608f84a98d7cf7c9e11b1a2b673cd23fc10dfc635ff2ad897"}
Jul 10 02:48:26.622630 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:26.622595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk" event={"ID":"27cf62e1-8629-41e0-a2ba-6909d779857e","Type":"ContainerStarted","Data":"214cb9d1a7b6472c5df454f808587e9838bc5f94a8511b2800810828f4d8ee62"}
Jul 10 02:48:26.639211 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:26.639160    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rz4wk" podStartSLOduration=2.299116089 podStartE2EDuration="26.639142289s" totalImagesPullingTime="24.3400262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:26.638176049 +0000 UTC m=+500.451254132" watchObservedRunningTime="2026-07-10 02:48:26.639142289 +0000 UTC m=+500.452220379"
Jul 10 02:48:58.777329 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:58.777250    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:48:58.849633 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:58.849597    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:48:58.849801 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:58.849698    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:48:58.852530 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:58.852506    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qr4k8\""
Jul 10 02:48:58.956003 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:58.955967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76stv\" (UniqueName: \"kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv\") pod \"authorino-58cd955675-b6bld\" (UID: \"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858\") " pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:48:59.020579 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.020542    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:48:59.023327 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.023309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:48:59.032284 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.032255    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:48:59.057020 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.056984    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76stv\" (UniqueName: \"kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv\") pod \"authorino-58cd955675-b6bld\" (UID: \"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858\") " pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:48:59.057020 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.057030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc\") pod \"authorino-75d556bd77-cxqnx\" (UID: \"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d\") " pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:48:59.065485 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.065459    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76stv\" (UniqueName: \"kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv\") pod \"authorino-58cd955675-b6bld\" (UID: \"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858\") " pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:48:59.157649 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.157614    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc\") pod \"authorino-75d556bd77-cxqnx\" (UID: \"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d\") " pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:48:59.158483 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.158470    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:48:59.166430 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.166408    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc\") pod \"authorino-75d556bd77-cxqnx\" (UID: \"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d\") " pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:48:59.290341 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.290314    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:48:59.292595 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:48:59.292570    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6fd17f2_1ccf_42d4_b8b1_a8db7d7a4858.slice/crio-ef13bddae036875e0385f2a92c300822bfb1124de7be27644367ce24df3efdd7 WatchSource:0}: Error finding container ef13bddae036875e0385f2a92c300822bfb1124de7be27644367ce24df3efdd7: Status 404 returned error can't find the container with id ef13bddae036875e0385f2a92c300822bfb1124de7be27644367ce24df3efdd7
Jul 10 02:48:59.332115 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.332054    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:48:59.462440 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.462415    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:48:59.464753 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:48:59.464725    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c45fd9d_d27c_49c5_a8ad_96da37c92a8d.slice/crio-ec39e6eef217a63d85e4d44b699b577c8a1b824fb211805126f9a0ca72b1a3e0 WatchSource:0}: Error finding container ec39e6eef217a63d85e4d44b699b577c8a1b824fb211805126f9a0ca72b1a3e0: Status 404 returned error can't find the container with id ec39e6eef217a63d85e4d44b699b577c8a1b824fb211805126f9a0ca72b1a3e0
Jul 10 02:48:59.738696 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.738658    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b6bld" event={"ID":"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858","Type":"ContainerStarted","Data":"ef13bddae036875e0385f2a92c300822bfb1124de7be27644367ce24df3efdd7"}
Jul 10 02:48:59.739694 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:48:59.739670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cxqnx" event={"ID":"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d","Type":"ContainerStarted","Data":"ec39e6eef217a63d85e4d44b699b577c8a1b824fb211805126f9a0ca72b1a3e0"}
Jul 10 02:49:03.766386 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:03.766346    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cxqnx" event={"ID":"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d","Type":"ContainerStarted","Data":"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"}
Jul 10 02:49:03.767810 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:03.767787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b6bld" event={"ID":"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858","Type":"ContainerStarted","Data":"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"}
Jul 10 02:49:03.781741 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:03.781691    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-cxqnx" podStartSLOduration=1.497994957 podStartE2EDuration="4.781676s" totalImagesPullingTime="3.283681043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:03.780719383 +0000 UTC m=+537.593797475" watchObservedRunningTime="2026-07-10 02:49:03.781676 +0000 UTC m=+537.594754090"
Jul 10 02:49:03.796913 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:03.796860    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-b6bld" podStartSLOduration=2.3431902669999998 podStartE2EDuration="5.796839708s" totalImagesPullingTime="3.453649441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:03.796173272 +0000 UTC m=+537.609251360" watchObservedRunningTime="2026-07-10 02:49:03.796839708 +0000 UTC m=+537.609917799"
Jul 10 02:49:03.808681 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:03.808647    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:49:05.775800 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:05.775757    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-b6bld" podUID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" containerName="authorino" containerID="cri-o://1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5" gracePeriod=30
Jul 10 02:49:06.020992 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.020966    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:49:06.118062 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.117960    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76stv\" (UniqueName: \"kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv\") pod \"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858\" (UID: \"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858\") "
Jul 10 02:49:06.120187 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.120161    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv" (OuterVolumeSpecName: "kube-api-access-76stv") pod "d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" (UID: "d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858"). InnerVolumeSpecName "kube-api-access-76stv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:49:06.219218 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.219175    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76stv\" (UniqueName: \"kubernetes.io/projected/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858-kube-api-access-76stv\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:49:06.780677 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.780638    2551 generic.go:356] "Generic (PLEG): container finished" podID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" containerID="1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5" exitCode=0
Jul 10 02:49:06.781067 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.780684    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b6bld"
Jul 10 02:49:06.781067 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.780699    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b6bld" event={"ID":"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858","Type":"ContainerDied","Data":"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"}
Jul 10 02:49:06.781067 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.780738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b6bld" event={"ID":"d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858","Type":"ContainerDied","Data":"ef13bddae036875e0385f2a92c300822bfb1124de7be27644367ce24df3efdd7"}
Jul 10 02:49:06.781067 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.780760    2551 scope.go:117] "RemoveContainer" containerID="1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"
Jul 10 02:49:06.788715 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.788695    2551 scope.go:117] "RemoveContainer" containerID="1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"
Jul 10 02:49:06.788988 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:49:06.788967    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5\": container with ID starting with 1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5 not found: ID does not exist" containerID="1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"
Jul 10 02:49:06.789047 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.788998    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5"} err="failed to get container status \"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5\": rpc error: code = NotFound desc = could not find container \"1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5\": container with ID starting with 1c0fc7e68a1d6a239475ce636c170ca0078326da6a6a323380dfe662aef646c5 not found: ID does not exist"
Jul 10 02:49:06.797491 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.797466    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:49:06.801393 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:06.801370    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-b6bld"]
Jul 10 02:49:08.686013 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:08.685979    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" path="/var/lib/kubelet/pods/d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858/volumes"
Jul 10 02:49:28.101816 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.101781    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:28.102209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.102069    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" containerName="authorino"
Jul 10 02:49:28.102209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.102082    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" containerName="authorino"
Jul 10 02:49:28.102209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.102160    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6fd17f2-1ccf-42d4-b8b1-a8db7d7a4858" containerName="authorino"
Jul 10 02:49:28.104723 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.104706    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:28.112209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.112183    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:28.194776 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.194738    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6msl\" (UniqueName: \"kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl\") pod \"authorino-5d4b67c446-zl2wv\" (UID: \"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579\") " pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:28.296151 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.296081    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6msl\" (UniqueName: \"kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl\") pod \"authorino-5d4b67c446-zl2wv\" (UID: \"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579\") " pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:28.304473 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.304438    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6msl\" (UniqueName: \"kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl\") pod \"authorino-5d4b67c446-zl2wv\" (UID: \"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579\") " pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:28.361674 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.361595    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:28.361850 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.361837    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:28.390938 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.390900    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:28.393811 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.393794    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:28.403017 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.402995    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:28.497541 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.497508    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdvh\" (UniqueName: \"kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh\") pod \"authorino-5f5d67c654-2ljsg\" (UID: \"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378\") " pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:28.497886 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.497867    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:28.500784 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:49:28.500741    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda74b5e78_a2c7_4cfe_a8bb_1b9438d62579.slice/crio-055ed4a278b70a71321b1846766276ef737eb56ce421d2cc1b500295fdcf129d WatchSource:0}: Error finding container 055ed4a278b70a71321b1846766276ef737eb56ce421d2cc1b500295fdcf129d: Status 404 returned error can't find the container with id 055ed4a278b70a71321b1846766276ef737eb56ce421d2cc1b500295fdcf129d
Jul 10 02:49:28.598954 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.598911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdvh\" (UniqueName: \"kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh\") pod \"authorino-5f5d67c654-2ljsg\" (UID: \"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378\") " pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:28.608570 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.608534    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdvh\" (UniqueName: \"kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh\") pod \"authorino-5f5d67c654-2ljsg\" (UID: \"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378\") " pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:28.706462 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.706428    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:28.831474 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.831444    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:28.834368 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:49:28.834339    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f6b8e1_a47b_4a33_a42d_7ed2d4a61378.slice/crio-5b5b43eefac3b15b04bfa5d5a7f3fc25e464da4daffec66d22832b779c0ea19d WatchSource:0}: Error finding container 5b5b43eefac3b15b04bfa5d5a7f3fc25e464da4daffec66d22832b779c0ea19d: Status 404 returned error can't find the container with id 5b5b43eefac3b15b04bfa5d5a7f3fc25e464da4daffec66d22832b779c0ea19d
Jul 10 02:49:28.871862 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.871819    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" event={"ID":"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579","Type":"ContainerStarted","Data":"055ed4a278b70a71321b1846766276ef737eb56ce421d2cc1b500295fdcf129d"}
Jul 10 02:49:28.872946 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:28.872917    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" event={"ID":"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378","Type":"ContainerStarted","Data":"5b5b43eefac3b15b04bfa5d5a7f3fc25e464da4daffec66d22832b779c0ea19d"}
Jul 10 02:49:29.879002 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.878956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" event={"ID":"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579","Type":"ContainerStarted","Data":"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"}
Jul 10 02:49:29.879555 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.879004    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" podUID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" containerName="authorino" containerID="cri-o://79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0" gracePeriod=30
Jul 10 02:49:29.880578 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.880554    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" event={"ID":"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378","Type":"ContainerStarted","Data":"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"}
Jul 10 02:49:29.917875 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.917825    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" podStartSLOduration=1.544768292 podStartE2EDuration="1.917807214s" totalImagesPullingTime="373.038922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:29.899848165 +0000 UTC m=+563.712926281" watchObservedRunningTime="2026-07-10 02:49:29.917807214 +0000 UTC m=+563.730885335"
Jul 10 02:49:29.918119 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.918079    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" podStartSLOduration=1.552513673 podStartE2EDuration="1.918071488s" totalImagesPullingTime="365.557815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:29.917399089 +0000 UTC m=+563.730477182" watchObservedRunningTime="2026-07-10 02:49:29.918071488 +0000 UTC m=+563.731149578"
Jul 10 02:49:29.945429 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.945393    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:49:29.945604 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:29.945581    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-cxqnx" podUID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" containerName="authorino" containerID="cri-o://d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f" gracePeriod=30
Jul 10 02:49:30.173196 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.173162    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:49:30.176399 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.176374    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.178844 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.178824    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 10 02:49:30.184318 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.184293    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:30.186125 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.186063    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:49:30.203369 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.203350    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:49:30.312447 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.312422    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc\") pod \"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d\" (UID: \"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d\") "
Jul 10 02:49:30.312576 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.312478    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6msl\" (UniqueName: \"kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl\") pod \"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579\" (UID: \"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579\") "
Jul 10 02:49:30.312576 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.312563    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n274\" (UniqueName: \"kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.312678 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.312661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.314521 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.314491    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc" (OuterVolumeSpecName: "kube-api-access-gc2vc") pod "6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" (UID: "6c45fd9d-d27c-49c5-a8ad-96da37c92a8d"). InnerVolumeSpecName "kube-api-access-gc2vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:49:30.314521 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.314503    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl" (OuterVolumeSpecName: "kube-api-access-p6msl") pod "a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" (UID: "a74b5e78-a2c7-4cfe-a8bb-1b9438d62579"). InnerVolumeSpecName "kube-api-access-p6msl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:49:30.413861 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.413812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n274\" (UniqueName: \"kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.414034 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.413989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.414086 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.414068    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d-kube-api-access-gc2vc\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:49:30.414156 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.414092    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6msl\" (UniqueName: \"kubernetes.io/projected/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579-kube-api-access-p6msl\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:49:30.416578 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.416532    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.422464 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.422437    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n274\" (UniqueName: \"kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274\") pod \"authorino-7c6b57448b-bs65j\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") " pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.490305 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.490272    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:49:30.620069 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.620037    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:49:30.622887 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:49:30.622854    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod890f0a7b_ae00_414e_aba8_d123a33de06a.slice/crio-863363d6f70e322c3c30b7b93986b020aaa65f80c2c5c03ea44f230248216a6a WatchSource:0}: Error finding container 863363d6f70e322c3c30b7b93986b020aaa65f80c2c5c03ea44f230248216a6a: Status 404 returned error can't find the container with id 863363d6f70e322c3c30b7b93986b020aaa65f80c2c5c03ea44f230248216a6a
Jul 10 02:49:30.886020 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.885982    2551 generic.go:356] "Generic (PLEG): container finished" podID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" containerID="79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0" exitCode=0
Jul 10 02:49:30.886020 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.886024    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zl2wv"
Jul 10 02:49:30.886567 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.886066    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" event={"ID":"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579","Type":"ContainerDied","Data":"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"}
Jul 10 02:49:30.886567 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.886118    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zl2wv" event={"ID":"a74b5e78-a2c7-4cfe-a8bb-1b9438d62579","Type":"ContainerDied","Data":"055ed4a278b70a71321b1846766276ef737eb56ce421d2cc1b500295fdcf129d"}
Jul 10 02:49:30.886567 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.886139    2551 scope.go:117] "RemoveContainer" containerID="79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"
Jul 10 02:49:30.887597 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.887402    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6b57448b-bs65j" event={"ID":"890f0a7b-ae00-414e-aba8-d123a33de06a","Type":"ContainerStarted","Data":"863363d6f70e322c3c30b7b93986b020aaa65f80c2c5c03ea44f230248216a6a"}
Jul 10 02:49:30.888935 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.888898    2551 generic.go:356] "Generic (PLEG): container finished" podID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" containerID="d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f" exitCode=0
Jul 10 02:49:30.888935 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.888935    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cxqnx"
Jul 10 02:49:30.889162 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.888952    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cxqnx" event={"ID":"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d","Type":"ContainerDied","Data":"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"}
Jul 10 02:49:30.889162 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.889009    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cxqnx" event={"ID":"6c45fd9d-d27c-49c5-a8ad-96da37c92a8d","Type":"ContainerDied","Data":"ec39e6eef217a63d85e4d44b699b577c8a1b824fb211805126f9a0ca72b1a3e0"}
Jul 10 02:49:30.894959 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.894943    2551 scope.go:117] "RemoveContainer" containerID="79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"
Jul 10 02:49:30.895251 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:49:30.895231    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0\": container with ID starting with 79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0 not found: ID does not exist" containerID="79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"
Jul 10 02:49:30.895346 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.895262    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0"} err="failed to get container status \"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0\": rpc error: code = NotFound desc = could not find container \"79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0\": container with ID starting with 79d4905d56514457f34164c774172bad0d5acb117f12f2a9e27d20c3443dbbe0 not found: ID does not exist"
Jul 10 02:49:30.895346 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.895286    2551 scope.go:117] "RemoveContainer" containerID="d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"
Jul 10 02:49:30.903014 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.902825    2551 scope.go:117] "RemoveContainer" containerID="d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"
Jul 10 02:49:30.903487 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:49:30.903468    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f\": container with ID starting with d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f not found: ID does not exist" containerID="d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"
Jul 10 02:49:30.903569 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.903494    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f"} err="failed to get container status \"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f\": rpc error: code = NotFound desc = could not find container \"d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f\": container with ID starting with d6151e763a9abf547402abe7ffb8e8df78188b02bcd17742f84fdb85a2ce7c5f not found: ID does not exist"
Jul 10 02:49:30.904438 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.904418    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:30.908555 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.908535    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zl2wv"]
Jul 10 02:49:30.922513 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.922488    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:49:30.926717 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:30.926693    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cxqnx"]
Jul 10 02:49:31.895210 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:31.895172    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6b57448b-bs65j" event={"ID":"890f0a7b-ae00-414e-aba8-d123a33de06a","Type":"ContainerStarted","Data":"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"}
Jul 10 02:49:31.913157 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:31.913088    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c6b57448b-bs65j" podStartSLOduration=1.573496086 podStartE2EDuration="1.913071921s" totalImagesPullingTime="339.575835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:31.912240408 +0000 UTC m=+565.725318476" watchObservedRunningTime="2026-07-10 02:49:31.913071921 +0000 UTC m=+565.726150012"
Jul 10 02:49:31.937767 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:31.937734    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:31.937915 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:31.937884    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" podUID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" containerName="authorino" containerID="cri-o://48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94" gracePeriod=30
Jul 10 02:49:32.192662 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.192637    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:32.329909 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.329871    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phdvh\" (UniqueName: \"kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh\") pod \"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378\" (UID: \"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378\") "
Jul 10 02:49:32.331962 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.331933    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh" (OuterVolumeSpecName: "kube-api-access-phdvh") pod "e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" (UID: "e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378"). InnerVolumeSpecName "kube-api-access-phdvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:49:32.431342 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.431248    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdvh\" (UniqueName: \"kubernetes.io/projected/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378-kube-api-access-phdvh\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:49:32.688782 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.688687    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" path="/var/lib/kubelet/pods/6c45fd9d-d27c-49c5-a8ad-96da37c92a8d/volumes"
Jul 10 02:49:32.689086 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.689067    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" path="/var/lib/kubelet/pods/a74b5e78-a2c7-4cfe-a8bb-1b9438d62579/volumes"
Jul 10 02:49:32.902204 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.902170    2551 generic.go:356] "Generic (PLEG): container finished" podID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" containerID="48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94" exitCode=0
Jul 10 02:49:32.902591 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.902225    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d67c654-2ljsg"
Jul 10 02:49:32.902591 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.902258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" event={"ID":"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378","Type":"ContainerDied","Data":"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"}
Jul 10 02:49:32.902591 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.902302    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d67c654-2ljsg" event={"ID":"e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378","Type":"ContainerDied","Data":"5b5b43eefac3b15b04bfa5d5a7f3fc25e464da4daffec66d22832b779c0ea19d"}
Jul 10 02:49:32.902591 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.902320    2551 scope.go:117] "RemoveContainer" containerID="48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"
Jul 10 02:49:32.910056 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.910036    2551 scope.go:117] "RemoveContainer" containerID="48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"
Jul 10 02:49:32.910322 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:49:32.910305    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94\": container with ID starting with 48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94 not found: ID does not exist" containerID="48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"
Jul 10 02:49:32.910403 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.910334    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94"} err="failed to get container status \"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94\": rpc error: code = NotFound desc = could not find container \"48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94\": container with ID starting with 48d78e7a4f1db18bf0119f1eb7d9bd364033dcb4011d477800d37e8cb1352e94 not found: ID does not exist"
Jul 10 02:49:32.918509 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.918481    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:32.922504 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:32.922485    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f5d67c654-2ljsg"]
Jul 10 02:49:34.686860 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.686829    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" path="/var/lib/kubelet/pods/e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378/volumes"
Jul 10 02:49:34.756787 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.756757    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757031    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757041    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757057    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757064    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757070    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757076    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" containerName="authorino"
Jul 10 02:49:34.757128 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757133    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c45fd9d-d27c-49c5-a8ad-96da37c92a8d" containerName="authorino"
Jul 10 02:49:34.757429 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757141    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f6b8e1-a47b-4a33-a42d-7ed2d4a61378" containerName="authorino"
Jul 10 02:49:34.757429 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.757148    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74b5e78-a2c7-4cfe-a8bb-1b9438d62579" containerName="authorino"
Jul 10 02:49:34.761325 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.761304    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.765227 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.765196    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 10 02:49:34.765227 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.765217    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 10 02:49:34.765668 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.765653    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5qr2q\""
Jul 10 02:49:34.771362 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.771341    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:49:34.847733 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.847695    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzrb\" (UniqueName: \"kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.847946 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.847745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.847946 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.847848    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.949140 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.949025    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzrb\" (UniqueName: \"kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.949140 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.949066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.949337 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.949206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.949808 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.949788    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.951542 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.951521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:34.958748 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:34.958725    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzrb\" (UniqueName: \"kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb\") pod \"maas-controller-7f9ff5cc88-h6nst\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") " pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:35.072197 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:35.072162    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:35.198869 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:35.198837    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:49:35.201785 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:49:35.201758    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de384d3_6af5_4147_b9d7_cf4ce9c85e29.slice/crio-9f5ae4bc8505186590cf165c68ce755011568cd0c78410c8967f4cfc3bfe77f6 WatchSource:0}: Error finding container 9f5ae4bc8505186590cf165c68ce755011568cd0c78410c8967f4cfc3bfe77f6: Status 404 returned error can't find the container with id 9f5ae4bc8505186590cf165c68ce755011568cd0c78410c8967f4cfc3bfe77f6
Jul 10 02:49:35.918474 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:35.918438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" event={"ID":"8de384d3-6af5-4147-b9d7-cf4ce9c85e29","Type":"ContainerStarted","Data":"9f5ae4bc8505186590cf165c68ce755011568cd0c78410c8967f4cfc3bfe77f6"}
Jul 10 02:49:37.928018 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:37.927979    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" event={"ID":"8de384d3-6af5-4147-b9d7-cf4ce9c85e29","Type":"ContainerStarted","Data":"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"}
Jul 10 02:49:37.928456 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:37.928124    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:49:37.948704 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:37.948649    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" podStartSLOduration=1.3579544399999999 podStartE2EDuration="3.948633017s" totalImagesPullingTime="2.590678577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:37.9470502 +0000 UTC m=+571.760128305" watchObservedRunningTime="2026-07-10 02:49:37.948633017 +0000 UTC m=+571.761711108"
Jul 10 02:49:48.938309 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:49:48.938276    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:50:06.539656 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:06.539626    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:50:06.540198 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:06.539637    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:50:40.262354 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.262283    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"]
Jul 10 02:50:40.270795 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.270767    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.273723 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.273695    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 10 02:50:40.274893 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.274867    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4zkwr\""
Jul 10 02:50:40.275028 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.274908    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 10 02:50:40.275285 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.275270    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 10 02:50:40.276177 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.276159    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"]
Jul 10 02:50:40.380015 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.379979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.380015 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.380017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.380268 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.380050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5rb\" (UniqueName: \"kubernetes.io/projected/ed8edd0b-f289-4a47-ba4b-f434fba398c1-kube-api-access-cs5rb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.380268 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.380126    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.380268 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.380169    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.380268 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.380203    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480672 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480682    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480735    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480759    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480786    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5rb\" (UniqueName: \"kubernetes.io/projected/ed8edd0b-f289-4a47-ba4b-f434fba398c1-kube-api-access-cs5rb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.480840 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.480824    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.481227 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.481198    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.481330 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.481264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.481330 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.481282    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.483067 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.483041    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8edd0b-f289-4a47-ba4b-f434fba398c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.483355 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.483337    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8edd0b-f289-4a47-ba4b-f434fba398c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.488974 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.488950    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5rb\" (UniqueName: \"kubernetes.io/projected/ed8edd0b-f289-4a47-ba4b-f434fba398c1-kube-api-access-cs5rb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb\" (UID: \"ed8edd0b-f289-4a47-ba4b-f434fba398c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.581842 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.581747    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:40.724057 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:40.724031    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"]
Jul 10 02:50:40.726892 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:50:40.726861    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8edd0b_f289_4a47_ba4b_f434fba398c1.slice/crio-4877af2f3c700ffa136f9922bdb397d491760c56bbe3ab2138526e7dd46aa2fc WatchSource:0}: Error finding container 4877af2f3c700ffa136f9922bdb397d491760c56bbe3ab2138526e7dd46aa2fc: Status 404 returned error can't find the container with id 4877af2f3c700ffa136f9922bdb397d491760c56bbe3ab2138526e7dd46aa2fc
Jul 10 02:50:41.198664 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:41.198592    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb" event={"ID":"ed8edd0b-f289-4a47-ba4b-f434fba398c1","Type":"ContainerStarted","Data":"4877af2f3c700ffa136f9922bdb397d491760c56bbe3ab2138526e7dd46aa2fc"}
Jul 10 02:50:42.204165 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:42.204049    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb" event={"ID":"ed8edd0b-f289-4a47-ba4b-f434fba398c1","Type":"ContainerStarted","Data":"3dac271d675d5580f67b94bda3bf4658f067e609d232dd498655f7b99a19f5b1"}
Jul 10 02:50:42.223720 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:42.223655    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb" podStartSLOduration=1.025562462 podStartE2EDuration="2.223635849s" totalImagesPullingTime="1.198073387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:50:42.221646011 +0000 UTC m=+636.034724126" watchObservedRunningTime="2026-07-10 02:50:42.223635849 +0000 UTC m=+636.036713958"
Jul 10 02:50:50.582085 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:50.582044    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:50.584522 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:50.582143    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:50.594803 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:50.594777    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:50:51.254011 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:50:51.253976    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb"
Jul 10 02:51:21.743517 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.743483    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"]
Jul 10 02:51:21.749309 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.749287    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.752040 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.752014    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 10 02:51:21.757731 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.757542    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"]
Jul 10 02:51:21.783876 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.783846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.783876 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.783877    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6gb\" (UniqueName: \"kubernetes.io/projected/06e2630d-9a39-48ce-b567-8e162cc03950-kube-api-access-fj6gb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.784082 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.783899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.784082 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.783953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.784082 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.783988    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.784082 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.784026    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e2630d-9a39-48ce-b567-8e162cc03950-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.884793 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884751    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.884793 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884803    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e2630d-9a39-48ce-b567-8e162cc03950-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885085 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885085 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884863    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6gb\" (UniqueName: \"kubernetes.io/projected/06e2630d-9a39-48ce-b567-8e162cc03950-kube-api-access-fj6gb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885085 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885085 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.884910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885321 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.885206    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885321 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.885258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.885321 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.885295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.887214 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.887193    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e2630d-9a39-48ce-b567-8e162cc03950-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.887390 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.887372    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e2630d-9a39-48ce-b567-8e162cc03950-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:21.893398 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:21.893374    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6gb\" (UniqueName: \"kubernetes.io/projected/06e2630d-9a39-48ce-b567-8e162cc03950-kube-api-access-fj6gb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5b6rx\" (UID: \"06e2630d-9a39-48ce-b567-8e162cc03950\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:22.061197 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:22.061071    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:22.206262 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:22.206231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"]
Jul 10 02:51:22.209438 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:51:22.209412    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e2630d_9a39_48ce_b567_8e162cc03950.slice/crio-9d0d773503631877c292b2a8c6de963afd6d946539be3a4027ce507496bc5cb9 WatchSource:0}: Error finding container 9d0d773503631877c292b2a8c6de963afd6d946539be3a4027ce507496bc5cb9: Status 404 returned error can't find the container with id 9d0d773503631877c292b2a8c6de963afd6d946539be3a4027ce507496bc5cb9
Jul 10 02:51:22.210757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:22.210739    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 02:51:22.364789 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:22.364714    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx" event={"ID":"06e2630d-9a39-48ce-b567-8e162cc03950","Type":"ContainerStarted","Data":"9d0d773503631877c292b2a8c6de963afd6d946539be3a4027ce507496bc5cb9"}
Jul 10 02:51:23.371948 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:23.371912    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx" event={"ID":"06e2630d-9a39-48ce-b567-8e162cc03950","Type":"ContainerStarted","Data":"67dc5369c0befb860b88b02ecbab4ca237e8ad3dc0b53eda15ebaf1696fa24f9"}
Jul 10 02:51:23.393983 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:23.393916    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx" podStartSLOduration=2.21610164 podStartE2EDuration="2.393897149s" totalImagesPullingTime="177.795509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:51:23.392006236 +0000 UTC m=+677.205084327" watchObservedRunningTime="2026-07-10 02:51:23.393897149 +0000 UTC m=+677.206975240"
Jul 10 02:51:32.061556 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:32.061512    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:32.061963 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:32.061578    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:32.074284 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:32.074251    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:51:32.418807 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:51:32.418772    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5b6rx"
Jul 10 02:52:24.745259 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.745170    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"]
Jul 10 02:52:24.748935 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.748819    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.751335 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.751306    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 10 02:52:24.757044 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.757017    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"]
Jul 10 02:52:24.804448 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804407    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.804448 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804450    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.804677 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804480    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.804677 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804555    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.804677 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.804677 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.804656    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slg79\" (UniqueName: \"kubernetes.io/projected/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-kube-api-access-slg79\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905189 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905151    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905380 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905204    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905380 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905236    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905380 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905262    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg79\" (UniqueName: \"kubernetes.io/projected/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-kube-api-access-slg79\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905380 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905380 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905682 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905657    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905760 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.905760 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.905737    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.907464 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.907442    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.907736 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.907718    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:24.912747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:24.912722    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slg79\" (UniqueName: \"kubernetes.io/projected/7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e-kube-api-access-slg79\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c\" (UID: \"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:25.059684 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:25.059594    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:25.193209 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:25.193184    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"]
Jul 10 02:52:25.195554 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:52:25.195527    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fb268d3_b3a0_4b2d_a2c0_1c7dc1f6940e.slice/crio-18f87ec47f0f589e2904706d09cc73bba7273fbf60ef1984590094aaf687de27 WatchSource:0}: Error finding container 18f87ec47f0f589e2904706d09cc73bba7273fbf60ef1984590094aaf687de27: Status 404 returned error can't find the container with id 18f87ec47f0f589e2904706d09cc73bba7273fbf60ef1984590094aaf687de27
Jul 10 02:52:25.626814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:25.626710    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c" event={"ID":"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e","Type":"ContainerStarted","Data":"357b278d44767d98db9f60545970d71c449c4d9b14902eb31ed5869e33fabd36"}
Jul 10 02:52:25.626814 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:25.626756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c" event={"ID":"7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e","Type":"ContainerStarted","Data":"18f87ec47f0f589e2904706d09cc73bba7273fbf60ef1984590094aaf687de27"}
Jul 10 02:52:25.645049 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:25.644993    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c" podStartSLOduration=1.4680474110000001 podStartE2EDuration="1.644976173s" totalImagesPullingTime="176.928762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:52:25.643541498 +0000 UTC m=+739.456619589" watchObservedRunningTime="2026-07-10 02:52:25.644976173 +0000 UTC m=+739.458054304"
Jul 10 02:52:35.060007 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:35.059949    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:35.060526 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:35.060024    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:35.072987 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:35.072957    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:52:35.680808 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:52:35.680775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c"
Jul 10 02:53:06.580898 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:06.580868    2551 scope.go:117] "RemoveContainer" containerID="9a872f3eb467be081b77b0527a2a0943e670f33cdc054bc9c009a1da4ae231d5"
Jul 10 02:53:36.535789 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.535710    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d45f67fff-p8sdh"]
Jul 10 02:53:36.538936 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.538919    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.546449 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.546428    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d45f67fff-p8sdh"]
Jul 10 02:53:36.615644 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.615606    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnzj\" (UniqueName: \"kubernetes.io/projected/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-kube-api-access-qnnzj\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.615856 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.615665    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-tls-cert\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.716162 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.716129    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnzj\" (UniqueName: \"kubernetes.io/projected/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-kube-api-access-qnnzj\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.716353 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.716188    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-tls-cert\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.718650 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.718625    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-tls-cert\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.724528 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.724501    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnzj\" (UniqueName: \"kubernetes.io/projected/754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f-kube-api-access-qnnzj\") pod \"authorino-6d45f67fff-p8sdh\" (UID: \"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f\") " pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.848842 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.848756    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d45f67fff-p8sdh"
Jul 10 02:53:36.987688 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:36.987664    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d45f67fff-p8sdh"]
Jul 10 02:53:36.990460 ip-10-0-136-90 kubenswrapper[2551]: W0710 02:53:36.990435    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754b3f0e_9bf9_4d7d_8ad2_6044ead62f0f.slice/crio-546096faeae3e9ea9ecb26cea5f0003ac537a17338f63b48ab7128e904e1df2f WatchSource:0}: Error finding container 546096faeae3e9ea9ecb26cea5f0003ac537a17338f63b48ab7128e904e1df2f: Status 404 returned error can't find the container with id 546096faeae3e9ea9ecb26cea5f0003ac537a17338f63b48ab7128e904e1df2f
Jul 10 02:53:37.926675 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:37.926640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d45f67fff-p8sdh" event={"ID":"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f","Type":"ContainerStarted","Data":"5d3fd79f3815686a7398bfe75cc32a467ecc16cd4f497c065422e81822decb3e"}
Jul 10 02:53:37.926675 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:37.926682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d45f67fff-p8sdh" event={"ID":"754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f","Type":"ContainerStarted","Data":"546096faeae3e9ea9ecb26cea5f0003ac537a17338f63b48ab7128e904e1df2f"}
Jul 10 02:53:37.968611 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:37.968533    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d45f67fff-p8sdh" podStartSLOduration=1.531114045 podStartE2EDuration="1.968507858s" totalImagesPullingTime="437.393813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:53:37.967538903 +0000 UTC m=+811.780616996" watchObservedRunningTime="2026-07-10 02:53:37.968507858 +0000 UTC m=+811.781585948"
Jul 10 02:53:38.026580 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.026532    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:53:38.026773 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.026753    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c6b57448b-bs65j" podUID="890f0a7b-ae00-414e-aba8-d123a33de06a" containerName="authorino" containerID="cri-o://89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d" gracePeriod=30
Jul 10 02:53:38.310029 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.310001    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:53:38.430063 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.430027    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n274\" (UniqueName: \"kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274\") pod \"890f0a7b-ae00-414e-aba8-d123a33de06a\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") "
Jul 10 02:53:38.430259 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.430073    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert\") pod \"890f0a7b-ae00-414e-aba8-d123a33de06a\" (UID: \"890f0a7b-ae00-414e-aba8-d123a33de06a\") "
Jul 10 02:53:38.432280 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.432240    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274" (OuterVolumeSpecName: "kube-api-access-5n274") pod "890f0a7b-ae00-414e-aba8-d123a33de06a" (UID: "890f0a7b-ae00-414e-aba8-d123a33de06a"). InnerVolumeSpecName "kube-api-access-5n274". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:53:38.440615 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.440578    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "890f0a7b-ae00-414e-aba8-d123a33de06a" (UID: "890f0a7b-ae00-414e-aba8-d123a33de06a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 02:53:38.531642 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.531535    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n274\" (UniqueName: \"kubernetes.io/projected/890f0a7b-ae00-414e-aba8-d123a33de06a-kube-api-access-5n274\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:53:38.531642 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.531564    2551 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/890f0a7b-ae00-414e-aba8-d123a33de06a-tls-cert\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:53:38.932290 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.932248    2551 generic.go:356] "Generic (PLEG): container finished" podID="890f0a7b-ae00-414e-aba8-d123a33de06a" containerID="89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d" exitCode=0
Jul 10 02:53:38.932747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.932318    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6b57448b-bs65j"
Jul 10 02:53:38.932747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.932329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6b57448b-bs65j" event={"ID":"890f0a7b-ae00-414e-aba8-d123a33de06a","Type":"ContainerDied","Data":"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"}
Jul 10 02:53:38.932747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.932375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6b57448b-bs65j" event={"ID":"890f0a7b-ae00-414e-aba8-d123a33de06a","Type":"ContainerDied","Data":"863363d6f70e322c3c30b7b93986b020aaa65f80c2c5c03ea44f230248216a6a"}
Jul 10 02:53:38.932747 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.932397    2551 scope.go:117] "RemoveContainer" containerID="89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"
Jul 10 02:53:38.940985 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.940959    2551 scope.go:117] "RemoveContainer" containerID="89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"
Jul 10 02:53:38.941358 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:53:38.941335    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d\": container with ID starting with 89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d not found: ID does not exist" containerID="89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"
Jul 10 02:53:38.941442 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.941366    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d"} err="failed to get container status \"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d\": rpc error: code = NotFound desc = could not find container \"89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d\": container with ID starting with 89928e729e7986b6593d3f5978c0c55bec9e93d1e9767471043c7315cd6edf7d not found: ID does not exist"
Jul 10 02:53:38.996904 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:38.996864    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:53:39.001626 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:39.001588    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c6b57448b-bs65j"]
Jul 10 02:53:40.688746 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:53:40.688708    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890f0a7b-ae00-414e-aba8-d123a33de06a" path="/var/lib/kubelet/pods/890f0a7b-ae00-414e-aba8-d123a33de06a/volumes"
Jul 10 02:55:01.449757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.449656    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:55:01.450371 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.449955    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" podUID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" containerName="manager" containerID="cri-o://c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983" gracePeriod=30
Jul 10 02:55:01.700448 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.700375    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:55:01.835757 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.835719    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert\") pod \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") "
Jul 10 02:55:01.835951 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.835780    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkzrb\" (UniqueName: \"kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb\") pod \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") "
Jul 10 02:55:01.835951 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.835825    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca\") pod \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\" (UID: \"8de384d3-6af5-4147-b9d7-cf4ce9c85e29\") "
Jul 10 02:55:01.836313 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.836284    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca" (OuterVolumeSpecName: "service-ca") pod "8de384d3-6af5-4147-b9d7-cf4ce9c85e29" (UID: "8de384d3-6af5-4147-b9d7-cf4ce9c85e29"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 02:55:01.838065 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.838045    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8de384d3-6af5-4147-b9d7-cf4ce9c85e29" (UID: "8de384d3-6af5-4147-b9d7-cf4ce9c85e29"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 02:55:01.838147 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.838059    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb" (OuterVolumeSpecName: "kube-api-access-qkzrb") pod "8de384d3-6af5-4147-b9d7-cf4ce9c85e29" (UID: "8de384d3-6af5-4147-b9d7-cf4ce9c85e29"). InnerVolumeSpecName "kube-api-access-qkzrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 02:55:01.936727 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.936674    2551 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-webhook-cert\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:55:01.936727 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.936734    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzrb\" (UniqueName: \"kubernetes.io/projected/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-kube-api-access-qkzrb\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:55:01.936932 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:01.936749    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8de384d3-6af5-4147-b9d7-cf4ce9c85e29-service-ca\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 02:55:02.284205 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.284166    2551 generic.go:356] "Generic (PLEG): container finished" podID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" containerID="c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983" exitCode=0
Jul 10 02:55:02.284381 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.284215    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst"
Jul 10 02:55:02.284381 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.284257    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" event={"ID":"8de384d3-6af5-4147-b9d7-cf4ce9c85e29","Type":"ContainerDied","Data":"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"}
Jul 10 02:55:02.284381 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.284311    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9ff5cc88-h6nst" event={"ID":"8de384d3-6af5-4147-b9d7-cf4ce9c85e29","Type":"ContainerDied","Data":"9f5ae4bc8505186590cf165c68ce755011568cd0c78410c8967f4cfc3bfe77f6"}
Jul 10 02:55:02.284381 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.284334    2551 scope.go:117] "RemoveContainer" containerID="c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"
Jul 10 02:55:02.292929 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.292911    2551 scope.go:117] "RemoveContainer" containerID="c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"
Jul 10 02:55:02.293236 ip-10-0-136-90 kubenswrapper[2551]: E0710 02:55:02.293213    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983\": container with ID starting with c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983 not found: ID does not exist" containerID="c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"
Jul 10 02:55:02.293326 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.293243    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983"} err="failed to get container status \"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983\": rpc error: code = NotFound desc = could not find container \"c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983\": container with ID starting with c366f748cca2fcf55aaed41a8117af807feb5a436457a53d1c449f112ad70983 not found: ID does not exist"
Jul 10 02:55:02.304052 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.303997    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:55:02.307410 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.307384    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9ff5cc88-h6nst"]
Jul 10 02:55:02.687447 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:02.687418    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" path="/var/lib/kubelet/pods/8de384d3-6af5-4147-b9d7-cf4ce9c85e29/volumes"
Jul 10 02:55:06.563673 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:06.563641    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 02:55:06.567285 ip-10-0-136-90 kubenswrapper[2551]: I0710 02:55:06.567262    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:00:06.593436 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:00:06.593404    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:00:06.595147 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:00:06.595121    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:05:06.615420 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:05:06.615392    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:05:06.619425 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:05:06.619404    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:10:06.638390 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:10:06.638361    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:10:06.643324 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:10:06.643299    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:15:06.661160 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:15:06.661131    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:15:06.666592 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:15:06.666569    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:18:10.451274 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451234    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451540    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890f0a7b-ae00-414e-aba8-d123a33de06a" containerName="authorino"
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451551    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="890f0a7b-ae00-414e-aba8-d123a33de06a" containerName="authorino"
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451559    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" containerName="manager"
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451564    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" containerName="manager"
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451625    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de384d3-6af5-4147-b9d7-cf4ce9c85e29" containerName="manager"
Jul 10 03:18:10.451875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.451632    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="890f0a7b-ae00-414e-aba8-d123a33de06a" containerName="authorino"
Jul 10 03:18:10.454678 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.454659    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.457451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.457415    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-57198657\"/\"openshift-service-ca.crt\""
Jul 10 03:18:10.457451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.457425    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-57198657\"/\"e2e-lifecycle-model-57198657-kserve-self-signed-certs\""
Jul 10 03:18:10.458677 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.458657    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-57198657\"/\"kube-root-ca.crt\""
Jul 10 03:18:10.458810 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.458682    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-57198657\"/\"default-dockercfg-h9cv7\""
Jul 10 03:18:10.463707 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.463678    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:18:10.533332 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.533531 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533344    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.533531 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533433    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.533531 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533511    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.533678 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533541    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t678\" (UniqueName: \"kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.533678 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.533575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634424 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634620 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634454    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634620 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t678\" (UniqueName: \"kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634620 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634506    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634791 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634640    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634791 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634890 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634845    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.634890 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.634864    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.635019 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.635002    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.636762 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.636740    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.637042 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.637024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.642995 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.642973    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t678\" (UniqueName: \"kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678\") pod \"e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") " pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.766241 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.766142    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:10.903138 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:10.903076    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:18:10.905116 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:18:10.905076    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42436939_677e_40b3_9c99_617d88b79e94.slice/crio-b0b1881d5bd088f3b9adca41d312dded8a5d407cb4d77b4e7b111158ad0ef9e1 WatchSource:0}: Error finding container b0b1881d5bd088f3b9adca41d312dded8a5d407cb4d77b4e7b111158ad0ef9e1: Status 404 returned error can't find the container with id b0b1881d5bd088f3b9adca41d312dded8a5d407cb4d77b4e7b111158ad0ef9e1
Jul 10 03:18:11.909221 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:11.909182    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" event={"ID":"42436939-677e-40b3-9c99-617d88b79e94","Type":"ContainerStarted","Data":"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"}
Jul 10 03:18:11.909221 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:11.909221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" event={"ID":"42436939-677e-40b3-9c99-617d88b79e94","Type":"ContainerStarted","Data":"b0b1881d5bd088f3b9adca41d312dded8a5d407cb4d77b4e7b111158ad0ef9e1"}
Jul 10 03:18:11.926501 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:11.926447    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" podStartSLOduration=1.926428628 podStartE2EDuration="1.926428628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:18:11.925448353 +0000 UTC m=+2285.738526465" watchObservedRunningTime="2026-07-10 03:18:11.926428628 +0000 UTC m=+2285.739506720"
Jul 10 03:18:20.767239 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:20.767193    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:20.767734 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:20.767261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:20.779763 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:20.779734    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:18:20.961932 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:18:20.961905    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:20:06.683838 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:20:06.683732    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:20:06.691357 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:20:06.691335    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:24:01.993943 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:01.993908    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:24:17.014133 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.014072    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" podUID="42436939-677e-40b3-9c99-617d88b79e94" containerName="main" containerID="cri-o://9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372" gracePeriod=45
Jul 10 03:24:17.258240 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.258214    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:24:17.353610 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353522    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.353610 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353564    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.353610 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353598    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.353813 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353651    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.353864 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353804    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.353864 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353812    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:24:17.353864 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353846    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t678\" (UniqueName: \"kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678\") pod \"42436939-677e-40b3-9c99-617d88b79e94\" (UID: \"42436939-677e-40b3-9c99-617d88b79e94\") "
Jul 10 03:24:17.354013 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.353910    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache" (OuterVolumeSpecName: "model-cache") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:24:17.354069 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.354022    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home" (OuterVolumeSpecName: "home") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:24:17.354181 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.354165    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:17.354242 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.354184    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:17.354242 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.354197    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:17.355825 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.355792    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm" (OuterVolumeSpecName: "dshm") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:24:17.355944 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.355839    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:24:17.355944 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.355900    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678" (OuterVolumeSpecName: "kube-api-access-4t678") pod "42436939-677e-40b3-9c99-617d88b79e94" (UID: "42436939-677e-40b3-9c99-617d88b79e94"). InnerVolumeSpecName "kube-api-access-4t678". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:24:17.401082 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.401043    2551 generic.go:356] "Generic (PLEG): container finished" podID="42436939-677e-40b3-9c99-617d88b79e94" containerID="9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372" exitCode=0
Jul 10 03:24:17.401082 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.401090    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"
Jul 10 03:24:17.401323 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.401092    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" event={"ID":"42436939-677e-40b3-9c99-617d88b79e94","Type":"ContainerDied","Data":"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"}
Jul 10 03:24:17.401323 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.401150    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8" event={"ID":"42436939-677e-40b3-9c99-617d88b79e94","Type":"ContainerDied","Data":"b0b1881d5bd088f3b9adca41d312dded8a5d407cb4d77b4e7b111158ad0ef9e1"}
Jul 10 03:24:17.401323 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.401174    2551 scope.go:117] "RemoveContainer" containerID="9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"
Jul 10 03:24:17.409900 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.409880    2551 scope.go:117] "RemoveContainer" containerID="9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"
Jul 10 03:24:17.410199 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:24:17.410179    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372\": container with ID starting with 9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372 not found: ID does not exist" containerID="9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"
Jul 10 03:24:17.410281 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.410207    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372"} err="failed to get container status \"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372\": rpc error: code = NotFound desc = could not find container \"9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372\": container with ID starting with 9f6358d9e5b9fe3dc2d3cb7518a6f9fa2b9c4b37514c7b88481a8e6169cc0372 not found: ID does not exist"
Jul 10 03:24:17.421168 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.421137    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:24:17.423555 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.423532    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-57198657/e2e-lifecycle-model-57198657-kserve-c6d5578dc-qzgf8"]
Jul 10 03:24:17.455362 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.455333    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t678\" (UniqueName: \"kubernetes.io/projected/42436939-677e-40b3-9c99-617d88b79e94-kube-api-access-4t678\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:17.455362 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.455359    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42436939-677e-40b3-9c99-617d88b79e94-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:17.455539 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:17.455374    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42436939-677e-40b3-9c99-617d88b79e94-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:24:18.688420 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:24:18.688382    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42436939-677e-40b3-9c99-617d88b79e94" path="/var/lib/kubelet/pods/42436939-677e-40b3-9c99-617d88b79e94/volumes"
Jul 10 03:25:06.705891 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:25:06.705776    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:25:06.713678 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:25:06.713658    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:26:18.051851 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.051810    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:26:18.052760 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.052268    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42436939-677e-40b3-9c99-617d88b79e94" containerName="main"
Jul 10 03:26:18.052760 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.052285    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="42436939-677e-40b3-9c99-617d88b79e94" containerName="main"
Jul 10 03:26:18.052760 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.052366    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="42436939-677e-40b3-9c99-617d88b79e94" containerName="main"
Jul 10 03:26:18.055301 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.055280    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.057825 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.057799    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7f792\"/\"openshift-service-ca.crt\""
Jul 10 03:26:18.058835 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.058817    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7f792\"/\"default-dockercfg-2dv7f\""
Jul 10 03:26:18.058946 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.058928    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7f792\"/\"test-model-e7f792-kserve-self-signed-certs\""
Jul 10 03:26:18.059014 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.058996    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7f792\"/\"kube-root-ca.crt\""
Jul 10 03:26:18.065137 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.065090    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:26:18.153984 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.153945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.153984 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.153994    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.154248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.154010    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.154248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.154135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.154248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.154206    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.154248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.154230    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7lq\" (UniqueName: \"kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255215 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255175    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255403 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255403 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255281    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7lq\" (UniqueName: \"kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255403 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255326    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255403 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255369    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255403 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255690 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255670    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255801 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.255858 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.255809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.257686 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.257658    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.257935 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.257918    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.263437 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.263416    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7lq\" (UniqueName: \"kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq\") pod \"test-model-e7f792-kserve-6b6d48bb89-8rr9w\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") " pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.364712 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.364610    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:18.495505 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.495462    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:26:18.497672 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:26:18.497647    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod698d1bdf_4db1_4853_8a84_8e0486dd331c.slice/crio-a595cc6aa1529302d3190f7192b6696046833ce94d7fb1d45be7238f7f0a342c WatchSource:0}: Error finding container a595cc6aa1529302d3190f7192b6696046833ce94d7fb1d45be7238f7f0a342c: Status 404 returned error can't find the container with id a595cc6aa1529302d3190f7192b6696046833ce94d7fb1d45be7238f7f0a342c
Jul 10 03:26:18.893777 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.893740    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" event={"ID":"698d1bdf-4db1-4853-8a84-8e0486dd331c","Type":"ContainerStarted","Data":"590407304ac30149e7d456262e98c9d297d8cf2b23c6dcf077a133651a51c465"}
Jul 10 03:26:18.894000 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.893785    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" event={"ID":"698d1bdf-4db1-4853-8a84-8e0486dd331c","Type":"ContainerStarted","Data":"a595cc6aa1529302d3190f7192b6696046833ce94d7fb1d45be7238f7f0a342c"}
Jul 10 03:26:18.912559 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:18.912498    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" podStartSLOduration=0.912476651 podStartE2EDuration="912.476651ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:26:18.911733268 +0000 UTC m=+2772.724811359" watchObservedRunningTime="2026-07-10 03:26:18.912476651 +0000 UTC m=+2772.725554743"
Jul 10 03:26:28.365358 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:28.365261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:28.365774 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:28.365574    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:28.377655 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:28.377626    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:28.946644 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:28.946612    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:26:39.749033 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.748996    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:26:39.752459 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.752438    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.756493 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.756471    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48f01f\"/\"default-dockercfg-tr9gr\""
Jul 10 03:26:39.756493 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.756490    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48f01f\"/\"openshift-service-ca.crt\""
Jul 10 03:26:39.756665 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.756471    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48f01f\"/\"test-model-48f01f-kserve-self-signed-certs\""
Jul 10 03:26:39.756665 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.756472    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48f01f\"/\"kube-root-ca.crt\""
Jul 10 03:26:39.761601 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.761575    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:26:39.836310 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836264    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.836310 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836311    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.836520 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836355    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.836520 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836376    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.836520 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgclm\" (UniqueName: \"kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.836617 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.836527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937520 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937475    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937520 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937693 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgclm\" (UniqueName: \"kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937693 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937584    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937693 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937693 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.937976 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937951    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.938041 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.937975    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.938041 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.938024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.939927 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.939897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.940052 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.940036    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:39.945982 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:39.945955    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgclm\" (UniqueName: \"kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm\") pod \"test-model-48f01f-kserve-6786b459ff-ms6gj\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") " pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:40.062352 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:40.062254    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:40.201314 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:40.201288    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:26:40.203771 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:26:40.203747    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc362e33c_4276_4dd2_bd20_dfced38cf3e7.slice/crio-4fbf2801d05bcc3a9c11eb3c7b02099db2153d498fdba57ad90064fe32cf93f5 WatchSource:0}: Error finding container 4fbf2801d05bcc3a9c11eb3c7b02099db2153d498fdba57ad90064fe32cf93f5: Status 404 returned error can't find the container with id 4fbf2801d05bcc3a9c11eb3c7b02099db2153d498fdba57ad90064fe32cf93f5
Jul 10 03:26:40.985183 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:40.985141    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" event={"ID":"c362e33c-4276-4dd2-bd20-dfced38cf3e7","Type":"ContainerStarted","Data":"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"}
Jul 10 03:26:40.985664 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:40.985193    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" event={"ID":"c362e33c-4276-4dd2-bd20-dfced38cf3e7","Type":"ContainerStarted","Data":"4fbf2801d05bcc3a9c11eb3c7b02099db2153d498fdba57ad90064fe32cf93f5"}
Jul 10 03:26:41.001676 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:41.001624    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" podStartSLOduration=2.001605364 podStartE2EDuration="2.001605364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:26:41.000306507 +0000 UTC m=+2794.813384620" watchObservedRunningTime="2026-07-10 03:26:41.001605364 +0000 UTC m=+2794.814683455"
Jul 10 03:26:50.062743 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:50.062694    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:50.063199 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:50.062757    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:50.075039 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:50.075002    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:26:51.037669 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:26:51.037635    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:27:21.946988 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.946953    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:27:21.950233 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.950211    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:21.954373 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.954342    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-0566e7\"/\"default-dockercfg-n97tz\""
Jul 10 03:27:21.954524 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.954343    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-0566e7\"/\"openshift-service-ca.crt\""
Jul 10 03:27:21.954524 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.954426    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-0566e7\"/\"kube-root-ca.crt\""
Jul 10 03:27:21.954524 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.954342    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-0566e7\"/\"test-model-0566e7-kserve-self-signed-certs\""
Jul 10 03:27:21.959145 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:21.959124    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:27:22.004642 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004609    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fg7\" (UniqueName: \"kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.004822 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004681    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.004822 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004724    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.004822 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.004990 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004852    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.004990 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.004903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.105862 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.105820    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106067 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.105870    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106067 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.105893    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106067 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.105923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106067 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.105956    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106291 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.106127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fg7\" (UniqueName: \"kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106423 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.106401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106423 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.106412    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.106533 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.106457    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.108279 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.108258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.108537 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.108520    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.115800 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.115776    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fg7\" (UniqueName: \"kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7\") pod \"test-model-0566e7-kserve-6ffc4f66f9-gk9gv\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") " pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.260502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.260407    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:22.393375 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:22.393348    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:27:22.395757 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:27:22.395720    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5b5294b_e759_455b_810e_f01c9b96f9cb.slice/crio-399b2a59ea00a818942baa0541faae14a4cd4d0d2a5dd9372a9692263b3563a2 WatchSource:0}: Error finding container 399b2a59ea00a818942baa0541faae14a4cd4d0d2a5dd9372a9692263b3563a2: Status 404 returned error can't find the container with id 399b2a59ea00a818942baa0541faae14a4cd4d0d2a5dd9372a9692263b3563a2
Jul 10 03:27:23.160653 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:23.160620    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" event={"ID":"b5b5294b-e759-455b-810e-f01c9b96f9cb","Type":"ContainerStarted","Data":"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"}
Jul 10 03:27:23.160653 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:23.160659    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" event={"ID":"b5b5294b-e759-455b-810e-f01c9b96f9cb","Type":"ContainerStarted","Data":"399b2a59ea00a818942baa0541faae14a4cd4d0d2a5dd9372a9692263b3563a2"}
Jul 10 03:27:23.177567 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:23.177512    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" podStartSLOduration=2.177494917 podStartE2EDuration="2.177494917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:27:23.175798854 +0000 UTC m=+2836.988876945" watchObservedRunningTime="2026-07-10 03:27:23.177494917 +0000 UTC m=+2836.990573007"
Jul 10 03:27:32.260729 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:32.260690    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:32.261252 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:32.260745    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:32.273381 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:32.273352    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:27:33.212910 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:27:33.212879    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:28:46.460399 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:28:46.460363    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:29:01.481799 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.481734    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" podUID="b5b5294b-e759-455b-810e-f01c9b96f9cb" containerName="main" containerID="cri-o://54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12" gracePeriod=45
Jul 10 03:29:01.735609 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.735536    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:29:01.761061 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761022    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761269 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761165    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761340 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761282    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761340 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761314    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8fg7\" (UniqueName: \"kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761447 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761346    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:01.761447 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761372    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761447 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761420    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm\") pod \"b5b5294b-e759-455b-810e-f01c9b96f9cb\" (UID: \"b5b5294b-e759-455b-810e-f01c9b96f9cb\") "
Jul 10 03:29:01.761587 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761482    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:01.761912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761721    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:01.761912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761742    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:01.761912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.761879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home" (OuterVolumeSpecName: "home") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:01.764114 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.764065    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm" (OuterVolumeSpecName: "dshm") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:01.764212 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.764188    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:29:01.764607 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.764583    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7" (OuterVolumeSpecName: "kube-api-access-z8fg7") pod "b5b5294b-e759-455b-810e-f01c9b96f9cb" (UID: "b5b5294b-e759-455b-810e-f01c9b96f9cb"). InnerVolumeSpecName "kube-api-access-z8fg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:29:01.862901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.862856    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fg7\" (UniqueName: \"kubernetes.io/projected/b5b5294b-e759-455b-810e-f01c9b96f9cb-kube-api-access-z8fg7\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:01.862901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.862894    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:01.862901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.862903    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b5294b-e759-455b-810e-f01c9b96f9cb-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:01.862901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:01.862911    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b5294b-e759-455b-810e-f01c9b96f9cb-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:02.573897 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.573858    2551 generic.go:356] "Generic (PLEG): container finished" podID="b5b5294b-e759-455b-810e-f01c9b96f9cb" containerID="54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12" exitCode=0
Jul 10 03:29:02.574372 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.573904    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"
Jul 10 03:29:02.574372 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.573944    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" event={"ID":"b5b5294b-e759-455b-810e-f01c9b96f9cb","Type":"ContainerDied","Data":"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"}
Jul 10 03:29:02.574372 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.573988    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv" event={"ID":"b5b5294b-e759-455b-810e-f01c9b96f9cb","Type":"ContainerDied","Data":"399b2a59ea00a818942baa0541faae14a4cd4d0d2a5dd9372a9692263b3563a2"}
Jul 10 03:29:02.574372 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.574009    2551 scope.go:117] "RemoveContainer" containerID="54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"
Jul 10 03:29:02.583011 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.582994    2551 scope.go:117] "RemoveContainer" containerID="54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"
Jul 10 03:29:02.583339 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:29:02.583321    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12\": container with ID starting with 54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12 not found: ID does not exist" containerID="54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"
Jul 10 03:29:02.583385 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.583349    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12"} err="failed to get container status \"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12\": rpc error: code = NotFound desc = could not find container \"54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12\": container with ID starting with 54766352ed0eb5264404e86703e741356282371f8974e38d63e65689c5158d12 not found: ID does not exist"
Jul 10 03:29:02.594244 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.594204    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:29:02.597996 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.597967    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-reuse-0566e7/test-model-0566e7-kserve-6ffc4f66f9-gk9gv"]
Jul 10 03:29:02.687114 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:02.687072    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b5294b-e759-455b-810e-f01c9b96f9cb" path="/var/lib/kubelet/pods/b5b5294b-e759-455b-810e-f01c9b96f9cb/volumes"
Jul 10 03:29:33.526982 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:33.526897    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:29:37.428708 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:37.428670    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:29:48.548011 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.547963    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" podUID="698d1bdf-4db1-4853-8a84-8e0486dd331c" containerName="main" containerID="cri-o://590407304ac30149e7d456262e98c9d297d8cf2b23c6dcf077a133651a51c465" gracePeriod=45
Jul 10 03:29:48.770580 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.770546    2551 generic.go:356] "Generic (PLEG): container finished" podID="698d1bdf-4db1-4853-8a84-8e0486dd331c" containerID="590407304ac30149e7d456262e98c9d297d8cf2b23c6dcf077a133651a51c465" exitCode=0
Jul 10 03:29:48.770742 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.770593    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" event={"ID":"698d1bdf-4db1-4853-8a84-8e0486dd331c","Type":"ContainerDied","Data":"590407304ac30149e7d456262e98c9d297d8cf2b23c6dcf077a133651a51c465"}
Jul 10 03:29:48.801775 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.801711    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:29:48.978586 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978554    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978622    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978658    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978681    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978709    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp7lq\" (UniqueName: \"kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq\") pod \"698d1bdf-4db1-4853-8a84-8e0486dd331c\" (UID: \"698d1bdf-4db1-4853-8a84-8e0486dd331c\") "
Jul 10 03:29:48.978999 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978871    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home" (OuterVolumeSpecName: "home") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:48.978999 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978923    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache" (OuterVolumeSpecName: "model-cache") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:48.978999 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.978955    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:48.979190 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.979020    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:48.979190 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.979032    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:48.981001 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.980978    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm" (OuterVolumeSpecName: "dshm") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:48.981090 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.980999    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq" (OuterVolumeSpecName: "kube-api-access-lp7lq") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "kube-api-access-lp7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:29:48.981090 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:48.981035    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "698d1bdf-4db1-4853-8a84-8e0486dd331c" (UID: "698d1bdf-4db1-4853-8a84-8e0486dd331c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:29:49.080354 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.080268    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:49.080354 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.080297    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698d1bdf-4db1-4853-8a84-8e0486dd331c-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:49.080354 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.080307    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698d1bdf-4db1-4853-8a84-8e0486dd331c-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:49.080354 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.080317    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7lq\" (UniqueName: \"kubernetes.io/projected/698d1bdf-4db1-4853-8a84-8e0486dd331c-kube-api-access-lp7lq\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:49.776371 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.776338    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"
Jul 10 03:29:49.776371 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.776367    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w" event={"ID":"698d1bdf-4db1-4853-8a84-8e0486dd331c","Type":"ContainerDied","Data":"a595cc6aa1529302d3190f7192b6696046833ce94d7fb1d45be7238f7f0a342c"}
Jul 10 03:29:49.776917 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.776418    2551 scope.go:117] "RemoveContainer" containerID="590407304ac30149e7d456262e98c9d297d8cf2b23c6dcf077a133651a51c465"
Jul 10 03:29:49.797201 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.797173    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:29:49.805444 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:49.803456    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e7f792/test-model-e7f792-kserve-6b6d48bb89-8rr9w"]
Jul 10 03:29:50.687465 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:50.687431    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698d1bdf-4db1-4853-8a84-8e0486dd331c" path="/var/lib/kubelet/pods/698d1bdf-4db1-4853-8a84-8e0486dd331c/volumes"
Jul 10 03:29:52.450466 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.450414    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" podUID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" containerName="main" containerID="cri-o://e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a" gracePeriod=45
Jul 10 03:29:52.703297 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.703220    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:29:52.791288 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.791252    2551 generic.go:356] "Generic (PLEG): container finished" podID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" containerID="e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a" exitCode=0
Jul 10 03:29:52.791471 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.791295    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"
Jul 10 03:29:52.791471 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.791335    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" event={"ID":"c362e33c-4276-4dd2-bd20-dfced38cf3e7","Type":"ContainerDied","Data":"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"}
Jul 10 03:29:52.791471 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.791382    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj" event={"ID":"c362e33c-4276-4dd2-bd20-dfced38cf3e7","Type":"ContainerDied","Data":"4fbf2801d05bcc3a9c11eb3c7b02099db2153d498fdba57ad90064fe32cf93f5"}
Jul 10 03:29:52.791471 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.791404    2551 scope.go:117] "RemoveContainer" containerID="e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"
Jul 10 03:29:52.799441 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.799424    2551 scope.go:117] "RemoveContainer" containerID="e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"
Jul 10 03:29:52.799719 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:29:52.799699    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a\": container with ID starting with e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a not found: ID does not exist" containerID="e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"
Jul 10 03:29:52.799774 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.799732    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a"} err="failed to get container status \"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a\": rpc error: code = NotFound desc = could not find container \"e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a\": container with ID starting with e72894824d2340f7610ab39cadc855ecfe07fe8b9731729e3f7b35ea7d1ccf3a not found: ID does not exist"
Jul 10 03:29:52.807177 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807156    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807281 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807207    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807281 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807228    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807281 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807259    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgclm\" (UniqueName: \"kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807286    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807327    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs\") pod \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\" (UID: \"c362e33c-4276-4dd2-bd20-dfced38cf3e7\") "
Jul 10 03:29:52.807498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807464    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home" (OuterVolumeSpecName: "home") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:52.807498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:52.807498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807492    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:52.807700 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807576    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:52.807700 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807595    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:52.807700 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.807609    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:52.809617 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.809586    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm" (OuterVolumeSpecName: "kube-api-access-lgclm") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "kube-api-access-lgclm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:29:52.809617 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.809598    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:29:52.809747 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.809674    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm" (OuterVolumeSpecName: "dshm") pod "c362e33c-4276-4dd2-bd20-dfced38cf3e7" (UID: "c362e33c-4276-4dd2-bd20-dfced38cf3e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:29:52.908317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.908263    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c362e33c-4276-4dd2-bd20-dfced38cf3e7-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:52.908317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.908309    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgclm\" (UniqueName: \"kubernetes.io/projected/c362e33c-4276-4dd2-bd20-dfced38cf3e7-kube-api-access-lgclm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:52.908317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:52.908320    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c362e33c-4276-4dd2-bd20-dfced38cf3e7-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:29:53.113300 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:53.113268    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:29:53.116367 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:53.116345    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-48f01f/test-model-48f01f-kserve-6786b459ff-ms6gj"]
Jul 10 03:29:54.686940 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:29:54.686908    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" path="/var/lib/kubelet/pods/c362e33c-4276-4dd2-bd20-dfced38cf3e7/volumes"
Jul 10 03:30:00.153322 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153276    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153600    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698d1bdf-4db1-4853-8a84-8e0486dd331c" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153613    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="698d1bdf-4db1-4853-8a84-8e0486dd331c" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153623    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b5294b-e759-455b-810e-f01c9b96f9cb" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153628    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b5294b-e759-455b-810e-f01c9b96f9cb" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153634    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153640    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153687    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c362e33c-4276-4dd2-bd20-dfced38cf3e7" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153695    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b5294b-e759-455b-810e-f01c9b96f9cb" containerName="main"
Jul 10 03:30:00.153723 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.153702    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="698d1bdf-4db1-4853-8a84-8e0486dd331c" containerName="main"
Jul 10 03:30:00.156965 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.156945    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"
Jul 10 03:30:00.159490 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.159468    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nhxqz\""
Jul 10 03:30:00.166648 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.166615    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:30:00.168061 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.168035    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"
Jul 10 03:30:00.303287 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.303259    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:30:00.306249 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:30:00.306221    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1138a947_7d17_4513_893e_2d1a8462cdaf.slice/crio-b5b8c0ce3ad031bf0345c9657db1464e43e93e5d844e9b01d9319a341ef3cddb WatchSource:0}: Error finding container b5b8c0ce3ad031bf0345c9657db1464e43e93e5d844e9b01d9319a341ef3cddb: Status 404 returned error can't find the container with id b5b8c0ce3ad031bf0345c9657db1464e43e93e5d844e9b01d9319a341ef3cddb
Jul 10 03:30:00.308142 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.308128    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 03:30:00.826779 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:00.826742    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" event={"ID":"1138a947-7d17-4513-893e-2d1a8462cdaf","Type":"ContainerStarted","Data":"b5b8c0ce3ad031bf0345c9657db1464e43e93e5d844e9b01d9319a341ef3cddb"}
Jul 10 03:30:03.847698 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:03.847602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" event={"ID":"1138a947-7d17-4513-893e-2d1a8462cdaf","Type":"ContainerStarted","Data":"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"}
Jul 10 03:30:03.863859 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:03.863804    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" podStartSLOduration=0.962643099 podStartE2EDuration="3.863787072s" totalImagesPullingTime="2.901143973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:30:03.862328576 +0000 UTC m=+2997.675406658" watchObservedRunningTime="2026-07-10 03:30:03.863787072 +0000 UTC m=+2997.676865163"
Jul 10 03:30:06.729030 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:06.728911    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:30:06.736236 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:06.736212    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:30:43.324919 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:43.324886    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:30:43.325498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:30:43.325084    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" podUID="1138a947-7d17-4513-893e-2d1a8462cdaf" containerName="cleanup" containerID="cri-o://6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c" gracePeriod=30
Jul 10 03:31:13.494428 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:13.494404    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"
Jul 10 03:31:14.143436 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.143405    2551 generic.go:356] "Generic (PLEG): container finished" podID="1138a947-7d17-4513-893e-2d1a8462cdaf" containerID="6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c" exitCode=137
Jul 10 03:31:14.143608 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.143453    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"
Jul 10 03:31:14.143608 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.143452    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" event={"ID":"1138a947-7d17-4513-893e-2d1a8462cdaf","Type":"ContainerDied","Data":"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"}
Jul 10 03:31:14.143608 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.143517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x" event={"ID":"1138a947-7d17-4513-893e-2d1a8462cdaf","Type":"ContainerDied","Data":"b5b8c0ce3ad031bf0345c9657db1464e43e93e5d844e9b01d9319a341ef3cddb"}
Jul 10 03:31:14.143608 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.143548    2551 scope.go:117] "RemoveContainer" containerID="6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"
Jul 10 03:31:14.151800 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.151781    2551 scope.go:117] "RemoveContainer" containerID="6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"
Jul 10 03:31:14.152064 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:31:14.152046    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c\": container with ID starting with 6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c not found: ID does not exist" containerID="6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"
Jul 10 03:31:14.152151 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.152072    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c"} err="failed to get container status \"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c\": rpc error: code = NotFound desc = could not find container \"6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c\": container with ID starting with 6b48a7fc8434918fa05517b00addbb9070d9521ac6c52478b39f2ea718bb904c not found: ID does not exist"
Jul 10 03:31:14.162889 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.162831    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:31:14.164812 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.164789    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f792-29727570-7g76x"]
Jul 10 03:31:14.686984 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:31:14.686949    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1138a947-7d17-4513-893e-2d1a8462cdaf" path="/var/lib/kubelet/pods/1138a947-7d17-4513-893e-2d1a8462cdaf/volumes"
Jul 10 03:32:11.200963 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.200929    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:11.201389 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.201229    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1138a947-7d17-4513-893e-2d1a8462cdaf" containerName="cleanup"
Jul 10 03:32:11.201389 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.201240    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1138a947-7d17-4513-893e-2d1a8462cdaf" containerName="cleanup"
Jul 10 03:32:11.201389 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.201293    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1138a947-7d17-4513-893e-2d1a8462cdaf" containerName="cleanup"
Jul 10 03:32:11.204986 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.204970    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:11.207556 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.207532    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9w849\""
Jul 10 03:32:11.211662 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.211636    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:11.253227 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.253186    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnswt\" (UniqueName: \"kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt\") pod \"kuadrant-operator-catalog-kmr4k\" (UID: \"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3\") " pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:11.354618 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.354580    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnswt\" (UniqueName: \"kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt\") pod \"kuadrant-operator-catalog-kmr4k\" (UID: \"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3\") " pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:11.361980 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.361958    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnswt\" (UniqueName: \"kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt\") pod \"kuadrant-operator-catalog-kmr4k\" (UID: \"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3\") " pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:11.515680 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.515583    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:11.652706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:11.652680    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:11.655169 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:32:11.655141    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce6c33d_5f9f_4539_a671_cf1f22f3e3b3.slice/crio-23a9782ad6e2da52ec288a8593c6ce6219e26303a1a30257ddaf5b49f9358b35 WatchSource:0}: Error finding container 23a9782ad6e2da52ec288a8593c6ce6219e26303a1a30257ddaf5b49f9358b35: Status 404 returned error can't find the container with id 23a9782ad6e2da52ec288a8593c6ce6219e26303a1a30257ddaf5b49f9358b35
Jul 10 03:32:12.380989 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:12.380948    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" event={"ID":"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3","Type":"ContainerStarted","Data":"23a9782ad6e2da52ec288a8593c6ce6219e26303a1a30257ddaf5b49f9358b35"}
Jul 10 03:32:14.392475 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.392443    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" event={"ID":"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3","Type":"ContainerStarted","Data":"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"}
Jul 10 03:32:14.410093 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.410042    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" podStartSLOduration=1.112194875 podStartE2EDuration="3.410011371s" totalImagesPullingTime="2.297816496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:32:14.408404674 +0000 UTC m=+3128.221482793" watchObservedRunningTime="2026-07-10 03:32:14.410011371 +0000 UTC m=+3128.223089463"
Jul 10 03:32:14.755697 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.755624    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:32:14.759048 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.759023    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.761766 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.761606    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9d01ec-serving-cert\""
Jul 10 03:32:14.761766 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.761638    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 10 03:32:14.761766 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.761610    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 10 03:32:14.762032 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.761967    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcjsl\""
Jul 10 03:32:14.769688 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.769662    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:32:14.884170 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.884130    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.884343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.884193    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.884343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.884227    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjbn\" (UniqueName: \"kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.984731 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.984686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.984916 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.984741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjbn\" (UniqueName: \"kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.984916 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.984783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.985492 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.985471    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.987137 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.987116    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:14.995740 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:14.995714    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjbn\" (UniqueName: \"kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn\") pod \"maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:15.071610 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:15.071516    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:15.209966 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:15.209934    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:32:15.212890 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:32:15.212860    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba14d48_0478_4267_a427_6f2cfe21e3fb.slice/crio-add4d2cfadc041a52ee42a9be3b23258ce4bfa7d85dab75ea70c59e2b7fec3e1 WatchSource:0}: Error finding container add4d2cfadc041a52ee42a9be3b23258ce4bfa7d85dab75ea70c59e2b7fec3e1: Status 404 returned error can't find the container with id add4d2cfadc041a52ee42a9be3b23258ce4bfa7d85dab75ea70c59e2b7fec3e1
Jul 10 03:32:15.398157 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:15.398117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" event={"ID":"4ba14d48-0478-4267-a427-6f2cfe21e3fb","Type":"ContainerStarted","Data":"add4d2cfadc041a52ee42a9be3b23258ce4bfa7d85dab75ea70c59e2b7fec3e1"}
Jul 10 03:32:17.408233 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:17.408195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" event={"ID":"4ba14d48-0478-4267-a427-6f2cfe21e3fb","Type":"ContainerStarted","Data":"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"}
Jul 10 03:32:17.408686 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:17.408343    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:17.427909 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:17.427843    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" podStartSLOduration=1.835971222 podStartE2EDuration="3.427826073s" totalImagesPullingTime="1.591854851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:32:17.425620148 +0000 UTC m=+3131.238698276" watchObservedRunningTime="2026-07-10 03:32:17.427826073 +0000 UTC m=+3131.240904163"
Jul 10 03:32:21.516211 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:21.516168    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:21.516641 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:21.516233    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:21.539583 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:21.539556    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:22.452134 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:22.452089    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:22.486518 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:22.486478    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:23.418426 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:23.418396    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:32:24.436710 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:24.436657    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" podUID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" containerName="registry-server" containerID="cri-o://0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6" gracePeriod=2
Jul 10 03:32:24.678269 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:24.678243    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:24.764764 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:24.764679    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnswt\" (UniqueName: \"kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt\") pod \"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3\" (UID: \"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3\") "
Jul 10 03:32:24.766744 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:24.766717    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt" (OuterVolumeSpecName: "kube-api-access-pnswt") pod "3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" (UID: "3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3"). InnerVolumeSpecName "kube-api-access-pnswt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:32:24.865675 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:24.865646    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnswt\" (UniqueName: \"kubernetes.io/projected/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3-kube-api-access-pnswt\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:32:25.442519 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.442487    2551 generic.go:356] "Generic (PLEG): container finished" podID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" containerID="0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6" exitCode=0
Jul 10 03:32:25.442965 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.442532    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k"
Jul 10 03:32:25.442965 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.442573    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" event={"ID":"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3","Type":"ContainerDied","Data":"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"}
Jul 10 03:32:25.442965 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.442623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmr4k" event={"ID":"3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3","Type":"ContainerDied","Data":"23a9782ad6e2da52ec288a8593c6ce6219e26303a1a30257ddaf5b49f9358b35"}
Jul 10 03:32:25.442965 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.442643    2551 scope.go:117] "RemoveContainer" containerID="0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"
Jul 10 03:32:25.457416 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.457397    2551 scope.go:117] "RemoveContainer" containerID="0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"
Jul 10 03:32:25.457712 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:32:25.457695    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6\": container with ID starting with 0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6 not found: ID does not exist" containerID="0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"
Jul 10 03:32:25.457762 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.457718    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6"} err="failed to get container status \"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6\": rpc error: code = NotFound desc = could not find container \"0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6\": container with ID starting with 0c536827e9bd745c0e01a39f2a09aa90ec071d01a433125557514857c88efba6 not found: ID does not exist"
Jul 10 03:32:25.467889 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.467863    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:25.474328 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:25.474305    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmr4k"]
Jul 10 03:32:26.687197 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:32:26.687163    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" path="/var/lib/kubelet/pods/3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3/volumes"
Jul 10 03:34:45.469325 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.469289    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:34:45.469883 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.469537    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" podUID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" containerName="maas-api" containerID="cri-o://80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481" gracePeriod=30
Jul 10 03:34:45.711612 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.711587    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:34:45.795863 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.795769    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls\") pod \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") "
Jul 10 03:34:45.795863 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.795810    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzjbn\" (UniqueName: \"kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn\") pod \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") "
Jul 10 03:34:45.795863 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.795836    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca\") pod \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\" (UID: \"4ba14d48-0478-4267-a427-6f2cfe21e3fb\") "
Jul 10 03:34:45.796304 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.796262    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ba14d48-0478-4267-a427-6f2cfe21e3fb" (UID: "4ba14d48-0478-4267-a427-6f2cfe21e3fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 03:34:45.798018 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.797994    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ba14d48-0478-4267-a427-6f2cfe21e3fb" (UID: "4ba14d48-0478-4267-a427-6f2cfe21e3fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:34:45.798111 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.798067    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn" (OuterVolumeSpecName: "kube-api-access-bzjbn") pod "4ba14d48-0478-4267-a427-6f2cfe21e3fb" (UID: "4ba14d48-0478-4267-a427-6f2cfe21e3fb"). InnerVolumeSpecName "kube-api-access-bzjbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:34:45.897374 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.897340    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba14d48-0478-4267-a427-6f2cfe21e3fb-maas-api-tls\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:34:45.897374 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.897369    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzjbn\" (UniqueName: \"kubernetes.io/projected/4ba14d48-0478-4267-a427-6f2cfe21e3fb-kube-api-access-bzjbn\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:34:45.897374 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:45.897379    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba14d48-0478-4267-a427-6f2cfe21e3fb-openshift-service-ca\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:34:46.028580 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.028543    2551 generic.go:356] "Generic (PLEG): container finished" podID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" containerID="80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481" exitCode=0
Jul 10 03:34:46.028777 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.028591    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" event={"ID":"4ba14d48-0478-4267-a427-6f2cfe21e3fb","Type":"ContainerDied","Data":"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"}
Jul 10 03:34:46.028777 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.028593    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"
Jul 10 03:34:46.028777 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.028632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm" event={"ID":"4ba14d48-0478-4267-a427-6f2cfe21e3fb","Type":"ContainerDied","Data":"add4d2cfadc041a52ee42a9be3b23258ce4bfa7d85dab75ea70c59e2b7fec3e1"}
Jul 10 03:34:46.028777 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.028653    2551 scope.go:117] "RemoveContainer" containerID="80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"
Jul 10 03:34:46.038805 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.038778    2551 scope.go:117] "RemoveContainer" containerID="80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"
Jul 10 03:34:46.039320 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:34:46.039297    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481\": container with ID starting with 80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481 not found: ID does not exist" containerID="80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"
Jul 10 03:34:46.039406 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.039328    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481"} err="failed to get container status \"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481\": rpc error: code = NotFound desc = could not find container \"80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481\": container with ID starting with 80b011c123d068a6b2cf509f2d7cb44346a502cf90d060455d6d22dfb0af1481 not found: ID does not exist"
Jul 10 03:34:46.052075 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.052009    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:34:46.058673 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.058647    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d01ec-fc7df79fc-k64mm"]
Jul 10 03:34:46.687355 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:46.687321    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" path="/var/lib/kubelet/pods/4ba14d48-0478-4267-a427-6f2cfe21e3fb/volumes"
Jul 10 03:34:55.542849 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.542808    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543156    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" containerName="maas-api"
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543167    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" containerName="maas-api"
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543187    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" containerName="registry-server"
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543195    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" containerName="registry-server"
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543243    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce6c33d-5f9f-4539-a671-cf1f22f3e3b3" containerName="registry-server"
Jul 10 03:34:55.543343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.543251    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba14d48-0478-4267-a427-6f2cfe21e3fb" containerName="maas-api"
Jul 10 03:34:55.546251 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.546227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.549014 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.548989    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcjsl\""
Jul 10 03:34:55.549297 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.549002    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 10 03:34:55.549297 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.549024    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a7b338-serving-cert\""
Jul 10 03:34:55.550134 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.550113    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 10 03:34:55.558861 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.558833    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:34:55.579812 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.579771    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdt2\" (UniqueName: \"kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.579994 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.579844    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.579994 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.579907    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.681204 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.681171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.681414 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.681233    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdt2\" (UniqueName: \"kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.681414 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.681268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.681913 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.681890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.683830 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.683807    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.688781 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.688760    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdt2\" (UniqueName: \"kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2\") pod \"maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.857165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.857048    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:55.992120 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:55.992076    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:34:55.994444 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:34:55.994415    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e45b2_749a_47ce_a287_a2665d834fab.slice/crio-a6f7fa510da25264c768dbbb80bc28d000bb3be9356737f48c8408acf4193d93 WatchSource:0}: Error finding container a6f7fa510da25264c768dbbb80bc28d000bb3be9356737f48c8408acf4193d93: Status 404 returned error can't find the container with id a6f7fa510da25264c768dbbb80bc28d000bb3be9356737f48c8408acf4193d93
Jul 10 03:34:56.069224 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:56.069181    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" event={"ID":"1d0e45b2-749a-47ce-a287-a2665d834fab","Type":"ContainerStarted","Data":"a6f7fa510da25264c768dbbb80bc28d000bb3be9356737f48c8408acf4193d93"}
Jul 10 03:34:57.075075 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:57.075040    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" event={"ID":"1d0e45b2-749a-47ce-a287-a2665d834fab","Type":"ContainerStarted","Data":"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"}
Jul 10 03:34:57.075501 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:57.075220    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:34:57.092793 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:34:57.092731    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" podStartSLOduration=1.530425753 podStartE2EDuration="2.092712823s" totalImagesPullingTime="562.28707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:34:57.091476816 +0000 UTC m=+3290.904554907" watchObservedRunningTime="2026-07-10 03:34:57.092712823 +0000 UTC m=+3290.905790914"
Jul 10 03:35:03.084225 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.084189    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:35:03.570634 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.570602    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:35:03.574063 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.574041    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.576878 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.576849    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-17c732-serving-cert\""
Jul 10 03:35:03.585328 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.585301    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:35:03.650012 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.649969    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.650248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.650045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.650248 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.650092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4wz\" (UniqueName: \"kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.751443 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.751406    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4wz\" (UniqueName: \"kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.751641 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.751459    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.751641 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.751529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.752580 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.752542    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.755226 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.755197    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.760126 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.760087    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4wz\" (UniqueName: \"kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz\") pod \"maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:03.889927 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:03.889891    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:04.030331 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:04.030305    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:35:04.032366 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:35:04.032335    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de4cbc0_f4cd_4240_bdee_b33865678b40.slice/crio-60a623a1f9ba214ac643b702acb28e7b07f5b726499125f7cdb1bb213bcc71fe WatchSource:0}: Error finding container 60a623a1f9ba214ac643b702acb28e7b07f5b726499125f7cdb1bb213bcc71fe: Status 404 returned error can't find the container with id 60a623a1f9ba214ac643b702acb28e7b07f5b726499125f7cdb1bb213bcc71fe
Jul 10 03:35:04.033712 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:04.033690    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 03:35:04.104683 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:04.104644    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" event={"ID":"9de4cbc0-f4cd-4240-bdee-b33865678b40","Type":"ContainerStarted","Data":"60a623a1f9ba214ac643b702acb28e7b07f5b726499125f7cdb1bb213bcc71fe"}
Jul 10 03:35:05.109811 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:05.109775    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" event={"ID":"9de4cbc0-f4cd-4240-bdee-b33865678b40","Type":"ContainerStarted","Data":"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"}
Jul 10 03:35:05.110239 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:05.109850    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:05.127037 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:05.126971    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" podStartSLOduration=1.7290626119999999 podStartE2EDuration="2.12695044s" totalImagesPullingTime="397.887828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:35:05.126782594 +0000 UTC m=+3298.939860687" watchObservedRunningTime="2026-07-10 03:35:05.12695044 +0000 UTC m=+3298.940028532"
Jul 10 03:35:06.751621 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:06.751518    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:35:06.765083 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:06.759112    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:35:11.119059 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:11.119026    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:35:29.045856 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.045776    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:35:29.048927 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.048900    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.052615 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.052593    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"openshift-service-ca.crt\""
Jul 10 03:35:29.052740 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.052592    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"auth-test-model-a7b338-kserve-self-signed-certs\""
Jul 10 03:35:29.052740 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.052593    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"default-dockercfg-hfx9j\""
Jul 10 03:35:29.052740 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.052599    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"kube-root-ca.crt\""
Jul 10 03:35:29.059221 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.059196    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:35:29.069502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069469    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr8w\" (UniqueName: \"kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.069502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069505    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.069726 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069522    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.069726 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069582    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.069726 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.069726 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.069658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170605 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170559    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170605 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170605    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170870 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170640    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr8w\" (UniqueName: \"kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170870 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170665    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170870 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.170870 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.170724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.171175 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.171150    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.171276 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.171154    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.171276 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.171231    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.172991 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.172967    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.173149 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.173132    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.184490 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.179677    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr8w\" (UniqueName: \"kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w\") pod \"auth-test-model-a7b338-kserve-8b4775b4c-pp57j\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") " pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.359490 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.359405    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:29.487139 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:29.487112    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:35:29.488915 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:35:29.488876    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd96b05_32d2_4d93_9e45_a56bfceb9df4.slice/crio-f7f717d288084196c5e68f0aeae58a0a68cd882aa0ffed38052341f7c6b44ecc WatchSource:0}: Error finding container f7f717d288084196c5e68f0aeae58a0a68cd882aa0ffed38052341f7c6b44ecc: Status 404 returned error can't find the container with id f7f717d288084196c5e68f0aeae58a0a68cd882aa0ffed38052341f7c6b44ecc
Jul 10 03:35:30.218151 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:30.218119    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" event={"ID":"bdd96b05-32d2-4d93-9e45-a56bfceb9df4","Type":"ContainerStarted","Data":"ed43a04de0094d2267f389efb7ad3d2032ee18d0abd0d52744aff239a3712a6b"}
Jul 10 03:35:30.218621 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:30.218162    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" event={"ID":"bdd96b05-32d2-4d93-9e45-a56bfceb9df4","Type":"ContainerStarted","Data":"f7f717d288084196c5e68f0aeae58a0a68cd882aa0ffed38052341f7c6b44ecc"}
Jul 10 03:35:30.234573 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:30.234520    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" podStartSLOduration=1.234504759 podStartE2EDuration="1.234504759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:35:30.23328706 +0000 UTC m=+3324.046365151" watchObservedRunningTime="2026-07-10 03:35:30.234504759 +0000 UTC m=+3324.047582871"
Jul 10 03:35:39.360451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:39.360396    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:39.360451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:39.360453    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:39.373135 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:39.373085    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:35:40.271806 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:35:40.271777    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:36:02.956497 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.956456    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:36:02.959794 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.959770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:02.963084 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.963061    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"kube-root-ca.crt\""
Jul 10 03:36:02.963234 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.963061    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"default-dockercfg-njcdp\""
Jul 10 03:36:02.963234 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.963091    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"openshift-service-ca.crt\""
Jul 10 03:36:02.963234 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.963093    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"auth-test-model-17c732-kserve-self-signed-certs\""
Jul 10 03:36:02.967950 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:02.967544    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:36:03.077318 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktmd\" (UniqueName: \"kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.077502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077326    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.077502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077351    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.077502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077393    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.077502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.077502 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.077452    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.177968 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.177932    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktmd\" (UniqueName: \"kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.177968 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.177973    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178280 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178280 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178037    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178280 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178070    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178280 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178093    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178498 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178474    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178561 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.178561 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.178526    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.180445 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.180419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.180647 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.180631    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.186193 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.186170    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktmd\" (UniqueName: \"kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd\") pod \"auth-test-model-17c732-kserve-59bfd755c9-kkkb7\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") " pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.270344 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.270236    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:03.404975 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:03.404951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:36:03.406923 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:36:03.406894    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab05f2f1_d208_497c_9785_aa6c8433f5c1.slice/crio-f628311576604719abf1473909cee1bd47d5c218b823a8cbbc23e770f1b1611c WatchSource:0}: Error finding container f628311576604719abf1473909cee1bd47d5c218b823a8cbbc23e770f1b1611c: Status 404 returned error can't find the container with id f628311576604719abf1473909cee1bd47d5c218b823a8cbbc23e770f1b1611c
Jul 10 03:36:04.365522 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:04.365487    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" event={"ID":"ab05f2f1-d208-497c-9785-aa6c8433f5c1","Type":"ContainerStarted","Data":"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"}
Jul 10 03:36:04.365522 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:04.365528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" event={"ID":"ab05f2f1-d208-497c-9785-aa6c8433f5c1","Type":"ContainerStarted","Data":"f628311576604719abf1473909cee1bd47d5c218b823a8cbbc23e770f1b1611c"}
Jul 10 03:36:04.382246 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:04.382195    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" podStartSLOduration=2.38217821 podStartE2EDuration="2.38217821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:36:04.380278817 +0000 UTC m=+3358.193356890" watchObservedRunningTime="2026-07-10 03:36:04.38217821 +0000 UTC m=+3358.195256300"
Jul 10 03:36:13.271205 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:13.271165    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:13.271716 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:13.271225    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:13.283618 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:13.283589    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:36:13.412237 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:36:13.412201    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:38:38.925693 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:38.925600    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:38:39.739048 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:39.739012    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:38:53.946945 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:53.946903    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" podUID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" containerName="main" containerID="cri-o://ed43a04de0094d2267f389efb7ad3d2032ee18d0abd0d52744aff239a3712a6b" gracePeriod=45
Jul 10 03:38:54.090244 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.090209    2551 generic.go:356] "Generic (PLEG): container finished" podID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" containerID="ed43a04de0094d2267f389efb7ad3d2032ee18d0abd0d52744aff239a3712a6b" exitCode=0
Jul 10 03:38:54.090473 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.090269    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" event={"ID":"bdd96b05-32d2-4d93-9e45-a56bfceb9df4","Type":"ContainerDied","Data":"ed43a04de0094d2267f389efb7ad3d2032ee18d0abd0d52744aff239a3712a6b"}
Jul 10 03:38:54.200217 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.200154    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:38:54.384356 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384316    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr8w\" (UniqueName: \"kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384532 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384379    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384532 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384407    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384532 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384447    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384532 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384478    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384532 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384519    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home\") pod \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\" (UID: \"bdd96b05-32d2-4d93-9e45-a56bfceb9df4\") "
Jul 10 03:38:54.384782 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384700    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:54.384782 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384715    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:54.384888 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.384866    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home" (OuterVolumeSpecName: "home") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:54.386656 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.386616    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w" (OuterVolumeSpecName: "kube-api-access-vsr8w") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "kube-api-access-vsr8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:38:54.386656 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.386627    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm" (OuterVolumeSpecName: "dshm") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:54.386838 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.386679    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdd96b05-32d2-4d93-9e45-a56bfceb9df4" (UID: "bdd96b05-32d2-4d93-9e45-a56bfceb9df4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485620    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485650    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485658    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485666    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485675    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr8w\" (UniqueName: \"kubernetes.io/projected/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-kube-api-access-vsr8w\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.485706 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.485683    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd96b05-32d2-4d93-9e45-a56bfceb9df4-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:54.759684 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:54.759590    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" podUID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" containerName="main" containerID="cri-o://c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c" gracePeriod=45
Jul 10 03:38:55.007622 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.007595    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:38:55.095770 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.095686    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"
Jul 10 03:38:55.095770 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.095687    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j" event={"ID":"bdd96b05-32d2-4d93-9e45-a56bfceb9df4","Type":"ContainerDied","Data":"f7f717d288084196c5e68f0aeae58a0a68cd882aa0ffed38052341f7c6b44ecc"}
Jul 10 03:38:55.095995 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.095809    2551 scope.go:117] "RemoveContainer" containerID="ed43a04de0094d2267f389efb7ad3d2032ee18d0abd0d52744aff239a3712a6b"
Jul 10 03:38:55.097152 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.097125    2551 generic.go:356] "Generic (PLEG): container finished" podID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" containerID="c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c" exitCode=0
Jul 10 03:38:55.097251 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.097171    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"
Jul 10 03:38:55.097251 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.097212    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" event={"ID":"ab05f2f1-d208-497c-9785-aa6c8433f5c1","Type":"ContainerDied","Data":"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"}
Jul 10 03:38:55.097251 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.097245    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7" event={"ID":"ab05f2f1-d208-497c-9785-aa6c8433f5c1","Type":"ContainerDied","Data":"f628311576604719abf1473909cee1bd47d5c218b823a8cbbc23e770f1b1611c"}
Jul 10 03:38:55.104366 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.104345    2551 scope.go:117] "RemoveContainer" containerID="c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"
Jul 10 03:38:55.111895 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.111872    2551 scope.go:117] "RemoveContainer" containerID="c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"
Jul 10 03:38:55.112232 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:38:55.112197    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c\": container with ID starting with c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c not found: ID does not exist" containerID="c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"
Jul 10 03:38:55.112276 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.112242    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c"} err="failed to get container status \"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c\": rpc error: code = NotFound desc = could not find container \"c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c\": container with ID starting with c8a01e0680ea59ede84a6ce9042fec7d4a6e8de7387f6f4193b46a946cee844c not found: ID does not exist"
Jul 10 03:38:55.120775 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.120739    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:38:55.120980 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.120959    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/auth-test-model-a7b338-kserve-8b4775b4c-pp57j"]
Jul 10 03:38:55.191771 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.191734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.191771 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.191777    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lktmd\" (UniqueName: \"kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.191976 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.191821    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.192024 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.191992    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:55.192077 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192057    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.192143 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192091    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.192195 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192143    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm\") pod \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\" (UID: \"ab05f2f1-d208-497c-9785-aa6c8433f5c1\") "
Jul 10 03:38:55.192195 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home" (OuterVolumeSpecName: "home") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:55.192339 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192313    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:55.192441 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192420    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.192483 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192450    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.192483 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.192468    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.194115 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.194062    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd" (OuterVolumeSpecName: "kube-api-access-lktmd") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "kube-api-access-lktmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:38:55.194227 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.194122    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:38:55.194227 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.194153    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm" (OuterVolumeSpecName: "dshm") pod "ab05f2f1-d208-497c-9785-aa6c8433f5c1" (UID: "ab05f2f1-d208-497c-9785-aa6c8433f5c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:38:55.293277 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.293238    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktmd\" (UniqueName: \"kubernetes.io/projected/ab05f2f1-d208-497c-9785-aa6c8433f5c1-kube-api-access-lktmd\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.293277 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.293271    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05f2f1-d208-497c-9785-aa6c8433f5c1-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.293277 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.293282    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05f2f1-d208-497c-9785-aa6c8433f5c1-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:38:55.417981 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.417940    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:38:55.422697 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:55.422669    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/auth-test-model-17c732-kserve-59bfd755c9-kkkb7"]
Jul 10 03:38:56.688327 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:56.688294    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" path="/var/lib/kubelet/pods/ab05f2f1-d208-497c-9785-aa6c8433f5c1/volumes"
Jul 10 03:38:56.688705 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:38:56.688688    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" path="/var/lib/kubelet/pods/bdd96b05-32d2-4d93-9e45-a56bfceb9df4/volumes"
Jul 10 03:39:34.050236 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050200    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050494    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" containerName="main"
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050504    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" containerName="main"
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050515    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" containerName="main"
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050522    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" containerName="main"
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050575    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd96b05-32d2-4d93-9e45-a56bfceb9df4" containerName="main"
Jul 10 03:39:34.050691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.050583    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab05f2f1-d208-497c-9785-aa6c8433f5c1" containerName="main"
Jul 10 03:39:34.053483 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.053465    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.057079 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.057046    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"sub-test-model-17c732-kserve-self-signed-certs\""
Jul 10 03:39:34.057079 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.057071    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"kube-root-ca.crt\""
Jul 10 03:39:34.057292 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.057162    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"openshift-service-ca.crt\""
Jul 10 03:39:34.057292 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.057209    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"default-dockercfg-njcdp\""
Jul 10 03:39:34.066792 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.066758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:39:34.133036 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133000    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.133240 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133063    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.133240 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.133240 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133192    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.133240 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.133397 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.133244    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddbk\" (UniqueName: \"kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234243 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234198    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234438 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234266    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234438 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234351    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234438 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234374    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234438 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234390    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234438 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234417    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddbk\" (UniqueName: \"kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234710 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234648    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234766 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234710    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.234823 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.234801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.236527 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.236506    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.236829 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.236807    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.242543 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.242521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddbk\" (UniqueName: \"kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk\") pod \"sub-test-model-17c732-kserve-7fdb8787d5-9k452\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") " pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.364529 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.364436    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:34.491531 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:34.491506    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:39:34.494137 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:39:34.494093    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1643de1_5664_4255_931b_d61009f24507.slice/crio-0ee33e4515822a31abfaff57aa544b66c2ab998117b889807e29cacc4506fc5d WatchSource:0}: Error finding container 0ee33e4515822a31abfaff57aa544b66c2ab998117b889807e29cacc4506fc5d: Status 404 returned error can't find the container with id 0ee33e4515822a31abfaff57aa544b66c2ab998117b889807e29cacc4506fc5d
Jul 10 03:39:35.265627 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:35.265593    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" event={"ID":"f1643de1-5664-4255-931b-d61009f24507","Type":"ContainerStarted","Data":"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"}
Jul 10 03:39:35.265627 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:35.265633    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" event={"ID":"f1643de1-5664-4255-931b-d61009f24507","Type":"ContainerStarted","Data":"0ee33e4515822a31abfaff57aa544b66c2ab998117b889807e29cacc4506fc5d"}
Jul 10 03:39:35.281953 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:35.281895    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" podStartSLOduration=1.281876518 podStartE2EDuration="1.281876518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:39:35.28051509 +0000 UTC m=+3569.093593192" watchObservedRunningTime="2026-07-10 03:39:35.281876518 +0000 UTC m=+3569.094954698"
Jul 10 03:39:44.365305 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:44.365258    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:44.365305 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:44.365316    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:44.377890 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:44.377857    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:39:45.317745 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:39:45.317713    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:40:06.774230 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:06.774119    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:40:06.783291 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:06.783269    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:40:44.175142 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:44.175085    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:40:59.194589 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.194518    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" podUID="f1643de1-5664-4255-931b-d61009f24507" containerName="main" containerID="cri-o://3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a" gracePeriod=45
Jul 10 03:40:59.441582 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.441553    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:40:59.483676 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483587    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.483676 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483665    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hddbk\" (UniqueName: \"kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.483884 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483718    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.483884 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.483884 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483759    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.483884 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483796    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home\") pod \"f1643de1-5664-4255-931b-d61009f24507\" (UID: \"f1643de1-5664-4255-931b-d61009f24507\") "
Jul 10 03:40:59.484122 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.483993    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:40:59.484191 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.484166    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home" (OuterVolumeSpecName: "home") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:40:59.484329 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.484305    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:40:59.485838 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.485808    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm" (OuterVolumeSpecName: "dshm") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:40:59.485838 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.485824    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk" (OuterVolumeSpecName: "kube-api-access-hddbk") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "kube-api-access-hddbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:40:59.485983 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.485865    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1643de1-5664-4255-931b-d61009f24507" (UID: "f1643de1-5664-4255-931b-d61009f24507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:40:59.585454 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585398    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1643de1-5664-4255-931b-d61009f24507-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.585454 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585445    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.585454 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585458    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.585454 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585468    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.585724 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585479    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1643de1-5664-4255-931b-d61009f24507-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.585724 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.585491    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hddbk\" (UniqueName: \"kubernetes.io/projected/f1643de1-5664-4255-931b-d61009f24507-kube-api-access-hddbk\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:40:59.616015 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.615983    2551 generic.go:356] "Generic (PLEG): container finished" podID="f1643de1-5664-4255-931b-d61009f24507" containerID="3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a" exitCode=0
Jul 10 03:40:59.616182 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.616030    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"
Jul 10 03:40:59.616182 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.616030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" event={"ID":"f1643de1-5664-4255-931b-d61009f24507","Type":"ContainerDied","Data":"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"}
Jul 10 03:40:59.616182 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.616072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452" event={"ID":"f1643de1-5664-4255-931b-d61009f24507","Type":"ContainerDied","Data":"0ee33e4515822a31abfaff57aa544b66c2ab998117b889807e29cacc4506fc5d"}
Jul 10 03:40:59.616182 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.616116    2551 scope.go:117] "RemoveContainer" containerID="3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"
Jul 10 03:40:59.624506 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.624483    2551 scope.go:117] "RemoveContainer" containerID="3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"
Jul 10 03:40:59.624772 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:40:59.624755    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a\": container with ID starting with 3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a not found: ID does not exist" containerID="3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"
Jul 10 03:40:59.624810 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.624784    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a"} err="failed to get container status \"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a\": rpc error: code = NotFound desc = could not find container \"3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a\": container with ID starting with 3b563bc74f3bbcb928fc00a46fd54617e1992e072c3ecc1ae99b9b73b022e58a not found: ID does not exist"
Jul 10 03:40:59.636854 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.636824    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:40:59.639784 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:40:59.639758    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/sub-test-model-17c732-kserve-7fdb8787d5-9k452"]
Jul 10 03:41:00.687247 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:00.687214    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1643de1-5664-4255-931b-d61009f24507" path="/var/lib/kubelet/pods/f1643de1-5664-4255-931b-d61009f24507/volumes"
Jul 10 03:41:07.450499 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.450466    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:41:07.450937 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.450885    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1643de1-5664-4255-931b-d61009f24507" containerName="main"
Jul 10 03:41:07.450937 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.450902    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1643de1-5664-4255-931b-d61009f24507" containerName="main"
Jul 10 03:41:07.451063 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.450978    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1643de1-5664-4255-931b-d61009f24507" containerName="main"
Jul 10 03:41:07.455581 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.455555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.458144 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.458119    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"kube-root-ca.crt\""
Jul 10 03:41:07.459537 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.459513    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"rate-test-model-a7b338-kserve-self-signed-certs\""
Jul 10 03:41:07.459801 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.459782    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"openshift-service-ca.crt\""
Jul 10 03:41:07.460383 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.459539    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7b338\"/\"default-dockercfg-hfx9j\""
Jul 10 03:41:07.461714 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.461692    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:41:07.554162 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554123    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.554341 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.554341 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.554341 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.554341 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt62d\" (UniqueName: \"kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.554479 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.554352    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.654865 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654832    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.654865 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654864    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655159 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654882    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655159 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654904    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt62d\" (UniqueName: \"kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655159 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655159 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.654996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655365 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.655282    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655418 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.655399    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.655461 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.655426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.657247 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.657226    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.657492 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.657478    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.663295 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.663274    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt62d\" (UniqueName: \"kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d\") pod \"rate-test-model-a7b338-kserve-8c98f9d5f-ntx45\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") " pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.766745 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.766642    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:07.907197 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:07.907162    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:41:07.910040 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:41:07.910015    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e276f1_d3f1_49ed_aebd_4c694afbd7a8.slice/crio-5e4a246b9d46339c54f1b0dabc0f461ce0f8bd53d204d4a867dc83583be7fa57 WatchSource:0}: Error finding container 5e4a246b9d46339c54f1b0dabc0f461ce0f8bd53d204d4a867dc83583be7fa57: Status 404 returned error can't find the container with id 5e4a246b9d46339c54f1b0dabc0f461ce0f8bd53d204d4a867dc83583be7fa57
Jul 10 03:41:08.660137 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:08.660087    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" event={"ID":"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8","Type":"ContainerStarted","Data":"e6ede396b668ba7663a441d36589975157d11b7cfd017cbd09fabdc9fc7b0445"}
Jul 10 03:41:08.660537 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:08.660149    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" event={"ID":"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8","Type":"ContainerStarted","Data":"5e4a246b9d46339c54f1b0dabc0f461ce0f8bd53d204d4a867dc83583be7fa57"}
Jul 10 03:41:08.676861 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:08.676809    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" podStartSLOduration=1.676793575 podStartE2EDuration="1.676793575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:41:08.676235424 +0000 UTC m=+3662.489313532" watchObservedRunningTime="2026-07-10 03:41:08.676793575 +0000 UTC m=+3662.489871665"
Jul 10 03:41:17.767302 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:17.767261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:17.767803 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:17.767349    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:17.780035 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:17.780003    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:18.712783 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:18.712750    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:41:24.349516 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.349481    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:41:24.352671 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.352652    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.355033 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.355006    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"openshift-service-ca.crt\""
Jul 10 03:41:24.355200 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.355120    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"kube-root-ca.crt\""
Jul 10 03:41:24.355955 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.355927    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"default-dockercfg-njcdp\""
Jul 10 03:41:24.356059 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.355973    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17c732\"/\"rate-test-model-17c732-kserve-self-signed-certs\""
Jul 10 03:41:24.362925 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.362896    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:41:24.507363 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507327    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.507363 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507363    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.507623 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507390    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bddk\" (UniqueName: \"kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.507623 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507504    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.507623 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507540    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.507722 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.507619    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609138 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609015    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bddk\" (UniqueName: \"kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609138 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609394 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609394 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609394 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609354    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609394 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609599 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609479    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609599 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609510    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.609599 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.609579    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.611680 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.611655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.611883 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.611868    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.617396 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.617372    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bddk\" (UniqueName: \"kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk\") pod \"rate-test-model-17c732-kserve-657b469c7f-dwqx8\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") " pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.664315 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.664261    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:24.797143 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:24.797114    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:41:24.799644 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:41:24.799604    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a1829d_a75f_423f_8fa9_41b69a23d3dc.slice/crio-a99afe6c6bd50afd7c0b34521cc61acc8597fc1385a7dd43ff6082f14602cc10 WatchSource:0}: Error finding container a99afe6c6bd50afd7c0b34521cc61acc8597fc1385a7dd43ff6082f14602cc10: Status 404 returned error can't find the container with id a99afe6c6bd50afd7c0b34521cc61acc8597fc1385a7dd43ff6082f14602cc10
Jul 10 03:41:25.732977 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:25.732933    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" event={"ID":"c8a1829d-a75f-423f-8fa9-41b69a23d3dc","Type":"ContainerStarted","Data":"bd9b87acebbdc338a596f6dadbd32f7b79d0b667444410ffad4561e439561228"}
Jul 10 03:41:25.733401 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:25.732986    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" event={"ID":"c8a1829d-a75f-423f-8fa9-41b69a23d3dc","Type":"ContainerStarted","Data":"a99afe6c6bd50afd7c0b34521cc61acc8597fc1385a7dd43ff6082f14602cc10"}
Jul 10 03:41:25.750026 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:25.749976    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" podStartSLOduration=1.749957626 podStartE2EDuration="1.749957626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:41:25.748436158 +0000 UTC m=+3679.561514250" watchObservedRunningTime="2026-07-10 03:41:25.749957626 +0000 UTC m=+3679.563035757"
Jul 10 03:41:34.665075 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:34.664987    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:34.665075 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:34.665040    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:34.677553 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:34.677524    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:41:34.787035 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:41:34.787008    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:42:15.840709 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:15.840675    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:42:16.780442 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:16.780407    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:42:30.860653 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:30.860609    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" podUID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" containerName="main" containerID="cri-o://e6ede396b668ba7663a441d36589975157d11b7cfd017cbd09fabdc9fc7b0445" gracePeriod=45
Jul 10 03:42:31.012772 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.012740    2551 generic.go:356] "Generic (PLEG): container finished" podID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" containerID="e6ede396b668ba7663a441d36589975157d11b7cfd017cbd09fabdc9fc7b0445" exitCode=0
Jul 10 03:42:31.012949 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.012816    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" event={"ID":"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8","Type":"ContainerDied","Data":"e6ede396b668ba7663a441d36589975157d11b7cfd017cbd09fabdc9fc7b0445"}
Jul 10 03:42:31.115763 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.115692    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:42:31.174063 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174021    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174262 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174076    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt62d\" (UniqueName: \"kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174262 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174132    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174262 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174160    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174262 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174197    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174262 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174221    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm\") pod \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\" (UID: \"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8\") "
Jul 10 03:42:31.174529 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174346    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:31.174529 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174389    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:31.174529 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home" (OuterVolumeSpecName: "home") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:31.174642 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174550    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.174642 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.174595    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.176318 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.176294    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d" (OuterVolumeSpecName: "kube-api-access-bt62d") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "kube-api-access-bt62d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:42:31.176399 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.176313    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm" (OuterVolumeSpecName: "dshm") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:31.176399 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.176354    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" (UID: "a7e276f1-d3f1-49ed-aebd-4c694afbd7a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:42:31.275123 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.275061    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt62d\" (UniqueName: \"kubernetes.io/projected/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-kube-api-access-bt62d\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.275123 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.275089    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.275123 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.275113    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.275123 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.275122    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:31.800771 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:31.800720    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" podUID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" containerName="main" containerID="cri-o://bd9b87acebbdc338a596f6dadbd32f7b79d0b667444410ffad4561e439561228" gracePeriod=45
Jul 10 03:42:32.018168 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.018141    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"
Jul 10 03:42:32.018589 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.018140    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45" event={"ID":"a7e276f1-d3f1-49ed-aebd-4c694afbd7a8","Type":"ContainerDied","Data":"5e4a246b9d46339c54f1b0dabc0f461ce0f8bd53d204d4a867dc83583be7fa57"}
Jul 10 03:42:32.018589 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.018272    2551 scope.go:117] "RemoveContainer" containerID="e6ede396b668ba7663a441d36589975157d11b7cfd017cbd09fabdc9fc7b0445"
Jul 10 03:42:32.020092 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.020065    2551 generic.go:356] "Generic (PLEG): container finished" podID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" containerID="bd9b87acebbdc338a596f6dadbd32f7b79d0b667444410ffad4561e439561228" exitCode=0
Jul 10 03:42:32.020208 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.020121    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" event={"ID":"c8a1829d-a75f-423f-8fa9-41b69a23d3dc","Type":"ContainerDied","Data":"bd9b87acebbdc338a596f6dadbd32f7b79d0b667444410ffad4561e439561228"}
Jul 10 03:42:32.040116 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.040072    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:42:32.046212 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.046189    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7b338/rate-test-model-a7b338-kserve-8c98f9d5f-ntx45"]
Jul 10 03:42:32.062418 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.062340    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:42:32.082343 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082311    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082525 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082358    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082525 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082383    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082525 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082445    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082525 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082482    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bddk\" (UniqueName: \"kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082765 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082558    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache\") pod \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\" (UID: \"c8a1829d-a75f-423f-8fa9-41b69a23d3dc\") "
Jul 10 03:42:32.082861 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.082834    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:32.083190 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.083156    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:32.083190 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.083171    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home" (OuterVolumeSpecName: "home") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:32.083449 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.083429    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-home\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.083521 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.083455    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tmp-dir\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.083521 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.083468    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-model-cache\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.084595 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.084565    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm" (OuterVolumeSpecName: "dshm") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 03:42:32.084722 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.084702    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:42:32.084884 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.084860    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk" (OuterVolumeSpecName: "kube-api-access-9bddk") pod "c8a1829d-a75f-423f-8fa9-41b69a23d3dc" (UID: "c8a1829d-a75f-423f-8fa9-41b69a23d3dc"). InnerVolumeSpecName "kube-api-access-9bddk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:42:32.184827 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.184781    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bddk\" (UniqueName: \"kubernetes.io/projected/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-kube-api-access-9bddk\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.184827 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.184819    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-tls-certs\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.184827 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.184833    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a1829d-a75f-423f-8fa9-41b69a23d3dc-dshm\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:32.687190 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:32.687158    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" path="/var/lib/kubelet/pods/a7e276f1-d3f1-49ed-aebd-4c694afbd7a8/volumes"
Jul 10 03:42:33.026854 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:33.026767    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"
Jul 10 03:42:33.026854 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:33.026777    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8" event={"ID":"c8a1829d-a75f-423f-8fa9-41b69a23d3dc","Type":"ContainerDied","Data":"a99afe6c6bd50afd7c0b34521cc61acc8597fc1385a7dd43ff6082f14602cc10"}
Jul 10 03:42:33.026854 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:33.026828    2551 scope.go:117] "RemoveContainer" containerID="bd9b87acebbdc338a596f6dadbd32f7b79d0b667444410ffad4561e439561228"
Jul 10 03:42:33.044665 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:33.044637    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:42:33.048449 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:33.048417    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17c732/rate-test-model-17c732-kserve-657b469c7f-dwqx8"]
Jul 10 03:42:34.687088 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:34.687051    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" path="/var/lib/kubelet/pods/c8a1829d-a75f-423f-8fa9-41b69a23d3dc/volumes"
Jul 10 03:42:48.400815 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.400776    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236789743024"]
Jul 10 03:42:48.401342 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401305    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" containerName="main"
Jul 10 03:42:48.401342 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401326    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" containerName="main"
Jul 10 03:42:48.401464 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401360    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" containerName="main"
Jul 10 03:42:48.401464 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401369    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" containerName="main"
Jul 10 03:42:48.401464 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401451    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a1829d-a75f-423f-8fa9-41b69a23d3dc" containerName="main"
Jul 10 03:42:48.401608 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.401466    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e276f1-d3f1-49ed-aebd-4c694afbd7a8" containerName="main"
Jul 10 03:42:48.405686 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.405662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:48.407846 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.407826    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-tf9rv\""
Jul 10 03:42:48.410809 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.410786    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236789743024"]
Jul 10 03:42:48.527271 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.527233    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfc7z\" (UniqueName: \"kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z\") pod \"test-curl-7760-140236789743024\" (UID: \"889ca02f-9704-48a2-8e25-fb98135d5380\") " pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:48.628543 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.628498    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfc7z\" (UniqueName: \"kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z\") pod \"test-curl-7760-140236789743024\" (UID: \"889ca02f-9704-48a2-8e25-fb98135d5380\") " pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:48.636216 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.636183    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfc7z\" (UniqueName: \"kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z\") pod \"test-curl-7760-140236789743024\" (UID: \"889ca02f-9704-48a2-8e25-fb98135d5380\") " pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:48.716008 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.715924    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:48.844047 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.844020    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236789743024"]
Jul 10 03:42:48.846613 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:42:48.846585    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod889ca02f_9704_48a2_8e25_fb98135d5380.slice/crio-112dee5b3d25a7cb76ef393cb75752c5caa859a7fd8ef3e7ea1d8b468ad8439b WatchSource:0}: Error finding container 112dee5b3d25a7cb76ef393cb75752c5caa859a7fd8ef3e7ea1d8b468ad8439b: Status 404 returned error can't find the container with id 112dee5b3d25a7cb76ef393cb75752c5caa859a7fd8ef3e7ea1d8b468ad8439b
Jul 10 03:42:48.847934 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:48.847919    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 03:42:49.097986 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:49.097897    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236789743024" event={"ID":"889ca02f-9704-48a2-8e25-fb98135d5380","Type":"ContainerStarted","Data":"112dee5b3d25a7cb76ef393cb75752c5caa859a7fd8ef3e7ea1d8b468ad8439b"}
Jul 10 03:42:50.103340 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:50.103302    2551 generic.go:356] "Generic (PLEG): container finished" podID="889ca02f-9704-48a2-8e25-fb98135d5380" containerID="210d8183c991859e293a25d8325ba84e792a3d0be766f02451f82c5cd45e8c4f" exitCode=0
Jul 10 03:42:50.103751 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:50.103388    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236789743024" event={"ID":"889ca02f-9704-48a2-8e25-fb98135d5380","Type":"ContainerDied","Data":"210d8183c991859e293a25d8325ba84e792a3d0be766f02451f82c5cd45e8c4f"}
Jul 10 03:42:51.265470 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.265443    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:51.300875 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.300849    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236789743024_889ca02f-9704-48a2-8e25-fb98135d5380/test-curl-7760-140236789743024/0.log"
Jul 10 03:42:51.327161 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.327127    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236789743024"]
Jul 10 03:42:51.330184 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.330163    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236789743024"]
Jul 10 03:42:51.352825 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.352797    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfc7z\" (UniqueName: \"kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z\") pod \"889ca02f-9704-48a2-8e25-fb98135d5380\" (UID: \"889ca02f-9704-48a2-8e25-fb98135d5380\") "
Jul 10 03:42:51.354904 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.354876    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z" (OuterVolumeSpecName: "kube-api-access-lfc7z") pod "889ca02f-9704-48a2-8e25-fb98135d5380" (UID: "889ca02f-9704-48a2-8e25-fb98135d5380"). InnerVolumeSpecName "kube-api-access-lfc7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:42:51.453624 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.453587    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfc7z\" (UniqueName: \"kubernetes.io/projected/889ca02f-9704-48a2-8e25-fb98135d5380-kube-api-access-lfc7z\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:51.469857 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.469828    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236789742896"]
Jul 10 03:42:51.470152 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.470137    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889ca02f-9704-48a2-8e25-fb98135d5380" containerName="test-curl-7760-140236789743024"
Jul 10 03:42:51.470201 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.470153    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="889ca02f-9704-48a2-8e25-fb98135d5380" containerName="test-curl-7760-140236789743024"
Jul 10 03:42:51.470236 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.470222    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="889ca02f-9704-48a2-8e25-fb98135d5380" containerName="test-curl-7760-140236789743024"
Jul 10 03:42:51.473060 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.473043    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:51.477330 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.477301    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236789742896"]
Jul 10 03:42:51.554110 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.554061    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkvk\" (UniqueName: \"kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk\") pod \"test-curl-7760-140236789742896\" (UID: \"396a8639-4796-4b82-966d-9f738ee7edba\") " pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:51.654545 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.654513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkvk\" (UniqueName: \"kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk\") pod \"test-curl-7760-140236789742896\" (UID: \"396a8639-4796-4b82-966d-9f738ee7edba\") " pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:51.662757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.662731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkvk\" (UniqueName: \"kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk\") pod \"test-curl-7760-140236789742896\" (UID: \"396a8639-4796-4b82-966d-9f738ee7edba\") " pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:51.783503 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.783467    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:51.913847 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:51.913689    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236789742896"]
Jul 10 03:42:51.916303 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:42:51.916273    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod396a8639_4796_4b82_966d_9f738ee7edba.slice/crio-0eb15d30cac47f33df658905a821c6c6a2f44d6dacba92365da7753071654140 WatchSource:0}: Error finding container 0eb15d30cac47f33df658905a821c6c6a2f44d6dacba92365da7753071654140: Status 404 returned error can't find the container with id 0eb15d30cac47f33df658905a821c6c6a2f44d6dacba92365da7753071654140
Jul 10 03:42:52.112180 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:52.112143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236789742896" event={"ID":"396a8639-4796-4b82-966d-9f738ee7edba","Type":"ContainerStarted","Data":"0eb15d30cac47f33df658905a821c6c6a2f44d6dacba92365da7753071654140"}
Jul 10 03:42:52.113288 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:52.113268    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112dee5b3d25a7cb76ef393cb75752c5caa859a7fd8ef3e7ea1d8b468ad8439b"
Jul 10 03:42:52.113393 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:52.113303    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:52.123705 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:42:52.123675    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7760-140236789743024\" is forbidden: User \"system:node:ip-10-0-136-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-90.ec2.internal' and this object" podUID="889ca02f-9704-48a2-8e25-fb98135d5380" pod="openshift-ingress/test-curl-7760-140236789743024"
Jul 10 03:42:52.686733 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:52.686703    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889ca02f-9704-48a2-8e25-fb98135d5380" path="/var/lib/kubelet/pods/889ca02f-9704-48a2-8e25-fb98135d5380/volumes"
Jul 10 03:42:53.118889 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:53.118806    2551 generic.go:356] "Generic (PLEG): container finished" podID="396a8639-4796-4b82-966d-9f738ee7edba" containerID="c799304b9fcd3b6df0a7a4d5ff7d5818626ae1a756691fc45f62edf47c175f5e" exitCode=0
Jul 10 03:42:53.118889 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:53.118868    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236789742896" event={"ID":"396a8639-4796-4b82-966d-9f738ee7edba","Type":"ContainerDied","Data":"c799304b9fcd3b6df0a7a4d5ff7d5818626ae1a756691fc45f62edf47c175f5e"}
Jul 10 03:42:54.257767 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.257735    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:54.280362 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.280330    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236789742896_396a8639-4796-4b82-966d-9f738ee7edba/test-curl-7760-140236789742896/0.log"
Jul 10 03:42:54.304389 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.304352    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236789742896"]
Jul 10 03:42:54.309211 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.309180    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236789742896"]
Jul 10 03:42:54.380217 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.380123    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrkvk\" (UniqueName: \"kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk\") pod \"396a8639-4796-4b82-966d-9f738ee7edba\" (UID: \"396a8639-4796-4b82-966d-9f738ee7edba\") "
Jul 10 03:42:54.382235 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.382205    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk" (OuterVolumeSpecName: "kube-api-access-nrkvk") pod "396a8639-4796-4b82-966d-9f738ee7edba" (UID: "396a8639-4796-4b82-966d-9f738ee7edba"). InnerVolumeSpecName "kube-api-access-nrkvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:42:54.479668 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.479633    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:54.479964 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.479951    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396a8639-4796-4b82-966d-9f738ee7edba" containerName="test-curl-7760-140236789742896"
Jul 10 03:42:54.480014 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.479965    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="396a8639-4796-4b82-966d-9f738ee7edba" containerName="test-curl-7760-140236789742896"
Jul 10 03:42:54.480046 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.480031    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="396a8639-4796-4b82-966d-9f738ee7edba" containerName="test-curl-7760-140236789742896"
Jul 10 03:42:54.481569 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.481547    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkvk\" (UniqueName: \"kubernetes.io/projected/396a8639-4796-4b82-966d-9f738ee7edba-kube-api-access-nrkvk\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:54.482893 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.482878    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:54.486495 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.486470    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:54.582314 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.582277    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrnq\" (UniqueName: \"kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq\") pod \"test-curl-7760-140236786707312\" (UID: \"c9fe21dc-11df-491e-ad13-dfeeac1eb090\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:54.683524 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.683496    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrnq\" (UniqueName: \"kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq\") pod \"test-curl-7760-140236786707312\" (UID: \"c9fe21dc-11df-491e-ad13-dfeeac1eb090\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:54.687401 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.687377    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396a8639-4796-4b82-966d-9f738ee7edba" path="/var/lib/kubelet/pods/396a8639-4796-4b82-966d-9f738ee7edba/volumes"
Jul 10 03:42:54.691415 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.691393    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrnq\" (UniqueName: \"kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq\") pod \"test-curl-7760-140236786707312\" (UID: \"c9fe21dc-11df-491e-ad13-dfeeac1eb090\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:54.793352 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.793319    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:54.922024 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:54.921942    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:54.924535 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:42:54.924513    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9fe21dc_11df_491e_ad13_dfeeac1eb090.slice/crio-c406a5f86aa93eb349d9ecc5eea8c343dd1e6ddeeed40ff5d6c2a4593a3383cb WatchSource:0}: Error finding container c406a5f86aa93eb349d9ecc5eea8c343dd1e6ddeeed40ff5d6c2a4593a3383cb: Status 404 returned error can't find the container with id c406a5f86aa93eb349d9ecc5eea8c343dd1e6ddeeed40ff5d6c2a4593a3383cb
Jul 10 03:42:55.130137 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:55.130079    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786707312" event={"ID":"c9fe21dc-11df-491e-ad13-dfeeac1eb090","Type":"ContainerStarted","Data":"c406a5f86aa93eb349d9ecc5eea8c343dd1e6ddeeed40ff5d6c2a4593a3383cb"}
Jul 10 03:42:55.131341 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:55.131322    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236789742896"
Jul 10 03:42:55.131455 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:55.131364    2551 scope.go:117] "RemoveContainer" containerID="c799304b9fcd3b6df0a7a4d5ff7d5818626ae1a756691fc45f62edf47c175f5e"
Jul 10 03:42:56.137286 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:56.137248    2551 generic.go:356] "Generic (PLEG): container finished" podID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" containerID="2339765d939d3759ab43c94fb24ebc5e12f36cb0442b716794cca45c140a1ca3" exitCode=0
Jul 10 03:42:56.137694 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:56.137340    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786707312" event={"ID":"c9fe21dc-11df-491e-ad13-dfeeac1eb090","Type":"ContainerDied","Data":"2339765d939d3759ab43c94fb24ebc5e12f36cb0442b716794cca45c140a1ca3"}
Jul 10 03:42:57.285060 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.285033    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.304151 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.304115    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236786707312_c9fe21dc-11df-491e-ad13-dfeeac1eb090/test-curl-7760-140236786707312/0.log"
Jul 10 03:42:57.326619 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.326586    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:57.329229 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.329205    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:57.405720 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.405623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmrnq\" (UniqueName: \"kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq\") pod \"c9fe21dc-11df-491e-ad13-dfeeac1eb090\" (UID: \"c9fe21dc-11df-491e-ad13-dfeeac1eb090\") "
Jul 10 03:42:57.407691 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.407664    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq" (OuterVolumeSpecName: "kube-api-access-fmrnq") pod "c9fe21dc-11df-491e-ad13-dfeeac1eb090" (UID: "c9fe21dc-11df-491e-ad13-dfeeac1eb090"). InnerVolumeSpecName "kube-api-access-fmrnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:42:57.459049 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.459008    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:57.459407 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.459388    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" containerName="test-curl-7760-140236786707312"
Jul 10 03:42:57.459490 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.459410    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" containerName="test-curl-7760-140236786707312"
Jul 10 03:42:57.459490 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.459485    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" containerName="test-curl-7760-140236786707312"
Jul 10 03:42:57.462436 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.462416    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.469886 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.469858    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:57.506433 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.506405    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrnq\" (UniqueName: \"kubernetes.io/projected/c9fe21dc-11df-491e-ad13-dfeeac1eb090-kube-api-access-fmrnq\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:42:57.607744 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.607699    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22qw\" (UniqueName: \"kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw\") pod \"test-curl-7760-140236786707312\" (UID: \"85592746-1ded-404d-833e-893ae8e38770\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.709128 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.709003    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22qw\" (UniqueName: \"kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw\") pod \"test-curl-7760-140236786707312\" (UID: \"85592746-1ded-404d-833e-893ae8e38770\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.716663 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.716640    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22qw\" (UniqueName: \"kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw\") pod \"test-curl-7760-140236786707312\" (UID: \"85592746-1ded-404d-833e-893ae8e38770\") " pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.771918 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.771883    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:57.905208 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:57.905183    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:42:57.907792 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:42:57.907765    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85592746_1ded_404d_833e_893ae8e38770.slice/crio-6da97dd30fe1b2f0ffc1f530f148b68c8edd5a6680906f177f8b138b31ff8cef WatchSource:0}: Error finding container 6da97dd30fe1b2f0ffc1f530f148b68c8edd5a6680906f177f8b138b31ff8cef: Status 404 returned error can't find the container with id 6da97dd30fe1b2f0ffc1f530f148b68c8edd5a6680906f177f8b138b31ff8cef
Jul 10 03:42:58.147757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:58.147722    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786707312" event={"ID":"85592746-1ded-404d-833e-893ae8e38770","Type":"ContainerStarted","Data":"6da97dd30fe1b2f0ffc1f530f148b68c8edd5a6680906f177f8b138b31ff8cef"}
Jul 10 03:42:58.148968 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:58.148947    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c406a5f86aa93eb349d9ecc5eea8c343dd1e6ddeeed40ff5d6c2a4593a3383cb"
Jul 10 03:42:58.148968 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:58.148959    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:42:58.159727 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:58.159694    2551 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7760-140236786707312" oldPodUID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" podUID="85592746-1ded-404d-833e-893ae8e38770"
Jul 10 03:42:58.687000 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:58.686961    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9fe21dc-11df-491e-ad13-dfeeac1eb090" path="/var/lib/kubelet/pods/c9fe21dc-11df-491e-ad13-dfeeac1eb090/volumes"
Jul 10 03:42:59.154491 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:59.154457    2551 generic.go:356] "Generic (PLEG): container finished" podID="85592746-1ded-404d-833e-893ae8e38770" containerID="5185889bed78dab748773a6b4848a80e45d02a020ba4471387554142cbf6b0d7" exitCode=0
Jul 10 03:42:59.154715 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:42:59.154547    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786707312" event={"ID":"85592746-1ded-404d-833e-893ae8e38770","Type":"ContainerDied","Data":"5185889bed78dab748773a6b4848a80e45d02a020ba4471387554142cbf6b0d7"}
Jul 10 03:43:00.300951 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.300921    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:43:00.320912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.320882    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236786707312_85592746-1ded-404d-833e-893ae8e38770/test-curl-7760-140236786707312/0.log"
Jul 10 03:43:00.346164 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.346129    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:43:00.349586 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.349556    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236786707312"]
Jul 10 03:43:00.433180 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.433062    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h22qw\" (UniqueName: \"kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw\") pod \"85592746-1ded-404d-833e-893ae8e38770\" (UID: \"85592746-1ded-404d-833e-893ae8e38770\") "
Jul 10 03:43:00.435169 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.435142    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw" (OuterVolumeSpecName: "kube-api-access-h22qw") pod "85592746-1ded-404d-833e-893ae8e38770" (UID: "85592746-1ded-404d-833e-893ae8e38770"). InnerVolumeSpecName "kube-api-access-h22qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:43:00.476621 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.476585    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:00.476901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.476889    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85592746-1ded-404d-833e-893ae8e38770" containerName="test-curl-7760-140236786707312"
Jul 10 03:43:00.476938 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.476904    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="85592746-1ded-404d-833e-893ae8e38770" containerName="test-curl-7760-140236786707312"
Jul 10 03:43:00.476969 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.476955    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="85592746-1ded-404d-833e-893ae8e38770" containerName="test-curl-7760-140236786707312"
Jul 10 03:43:00.479780 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.479764    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:00.483804 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.483775    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:00.533890 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.533851    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22qw\" (UniqueName: \"kubernetes.io/projected/85592746-1ded-404d-833e-893ae8e38770-kube-api-access-h22qw\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:43:00.634581 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.634540    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dps\" (UniqueName: \"kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps\") pod \"test-curl-7760-140236786706448\" (UID: \"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:00.687430 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.687349    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85592746-1ded-404d-833e-893ae8e38770" path="/var/lib/kubelet/pods/85592746-1ded-404d-833e-893ae8e38770/volumes"
Jul 10 03:43:00.735658 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.735622    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dps\" (UniqueName: \"kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps\") pod \"test-curl-7760-140236786706448\" (UID: \"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:00.743814 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.743784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dps\" (UniqueName: \"kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps\") pod \"test-curl-7760-140236786706448\" (UID: \"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:00.789929 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.789892    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:00.921791 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:00.921598    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:00.924341 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:43:00.924299    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8effa8b_cbee_4601_9ab4_c0e0172bd3f1.slice/crio-b9ca9c55d7af61c55d79120f323b586e65c22ba8a9437b987dfd26ccc95681da WatchSource:0}: Error finding container b9ca9c55d7af61c55d79120f323b586e65c22ba8a9437b987dfd26ccc95681da: Status 404 returned error can't find the container with id b9ca9c55d7af61c55d79120f323b586e65c22ba8a9437b987dfd26ccc95681da
Jul 10 03:43:01.164811 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:01.164777    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786707312"
Jul 10 03:43:01.164811 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:01.164794    2551 scope.go:117] "RemoveContainer" containerID="5185889bed78dab748773a6b4848a80e45d02a020ba4471387554142cbf6b0d7"
Jul 10 03:43:01.166177 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:01.165931    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786706448" event={"ID":"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1","Type":"ContainerStarted","Data":"b9ca9c55d7af61c55d79120f323b586e65c22ba8a9437b987dfd26ccc95681da"}
Jul 10 03:43:02.171804 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:02.171772    2551 generic.go:356] "Generic (PLEG): container finished" podID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" containerID="0c8baac23e731c55994e7b9c40f1c24a23cb1294f2808cab72ef223e57a3b990" exitCode=0
Jul 10 03:43:02.172320 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:02.171872    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786706448" event={"ID":"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1","Type":"ContainerDied","Data":"0c8baac23e731c55994e7b9c40f1c24a23cb1294f2808cab72ef223e57a3b990"}
Jul 10 03:43:03.314301 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.314277    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.333429 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.333395    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236786706448_b8effa8b-cbee-4601-9ab4-c0e0172bd3f1/test-curl-7760-140236786706448/0.log"
Jul 10 03:43:03.355792 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.355761    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:03.359291 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.359265    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:03.457979 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.457883    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74dps\" (UniqueName: \"kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps\") pod \"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1\" (UID: \"b8effa8b-cbee-4601-9ab4-c0e0172bd3f1\") "
Jul 10 03:43:03.459985 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.459963    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps" (OuterVolumeSpecName: "kube-api-access-74dps") pod "b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" (UID: "b8effa8b-cbee-4601-9ab4-c0e0172bd3f1"). InnerVolumeSpecName "kube-api-access-74dps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:43:03.484971 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.482690    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:03.484971 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.483431    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" containerName="test-curl-7760-140236786706448"
Jul 10 03:43:03.484971 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.483451    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" containerName="test-curl-7760-140236786706448"
Jul 10 03:43:03.484971 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.483602    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" containerName="test-curl-7760-140236786706448"
Jul 10 03:43:03.489131 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.489089    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.492491 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.492464    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:03.558709 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.558672    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74dps\" (UniqueName: \"kubernetes.io/projected/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1-kube-api-access-74dps\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:43:03.659607 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.659560    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sxc\" (UniqueName: \"kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc\") pod \"test-curl-7760-140236786706448\" (UID: \"69358156-ba3f-4e15-9685-424041372e56\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.760868 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.760777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sxc\" (UniqueName: \"kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc\") pod \"test-curl-7760-140236786706448\" (UID: \"69358156-ba3f-4e15-9685-424041372e56\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.768324 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.768294    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sxc\" (UniqueName: \"kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc\") pod \"test-curl-7760-140236786706448\" (UID: \"69358156-ba3f-4e15-9685-424041372e56\") " pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.799501 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.799467    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:03.927032 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:03.927005    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:03.929612 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:43:03.929588    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69358156_ba3f_4e15_9685_424041372e56.slice/crio-5c973fc037a5dbeed2d0ac236a0eae656fb9da662b6362b70410c6e4ea7450cc WatchSource:0}: Error finding container 5c973fc037a5dbeed2d0ac236a0eae656fb9da662b6362b70410c6e4ea7450cc: Status 404 returned error can't find the container with id 5c973fc037a5dbeed2d0ac236a0eae656fb9da662b6362b70410c6e4ea7450cc
Jul 10 03:43:04.182360 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:04.182332    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ca9c55d7af61c55d79120f323b586e65c22ba8a9437b987dfd26ccc95681da"
Jul 10 03:43:04.182556 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:04.182361    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:04.183491 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:04.183468    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786706448" event={"ID":"69358156-ba3f-4e15-9685-424041372e56","Type":"ContainerStarted","Data":"5c973fc037a5dbeed2d0ac236a0eae656fb9da662b6362b70410c6e4ea7450cc"}
Jul 10 03:43:04.194194 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:04.194165    2551 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7760-140236786706448" oldPodUID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" podUID="69358156-ba3f-4e15-9685-424041372e56"
Jul 10 03:43:04.686485 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:04.686457    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8effa8b-cbee-4601-9ab4-c0e0172bd3f1" path="/var/lib/kubelet/pods/b8effa8b-cbee-4601-9ab4-c0e0172bd3f1/volumes"
Jul 10 03:43:05.189282 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:05.189246    2551 generic.go:356] "Generic (PLEG): container finished" podID="69358156-ba3f-4e15-9685-424041372e56" containerID="7cce0f897c08e7f638a4f02c38efa4d5b5493be7f171ed0828d403815efaa73f" exitCode=0
Jul 10 03:43:05.189460 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:05.189295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7760-140236786706448" event={"ID":"69358156-ba3f-4e15-9685-424041372e56","Type":"ContainerDied","Data":"7cce0f897c08e7f638a4f02c38efa4d5b5493be7f171ed0828d403815efaa73f"}
Jul 10 03:43:06.334839 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.334812    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:06.357046 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.357021    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7760-140236786706448_69358156-ba3f-4e15-9685-424041372e56/test-curl-7760-140236786706448/0.log"
Jul 10 03:43:06.379726 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.379690    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:06.383015 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.382983    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7760-140236786706448"]
Jul 10 03:43:06.480774 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.480679    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6sxc\" (UniqueName: \"kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc\") pod \"69358156-ba3f-4e15-9685-424041372e56\" (UID: \"69358156-ba3f-4e15-9685-424041372e56\") "
Jul 10 03:43:06.482771 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.482737    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc" (OuterVolumeSpecName: "kube-api-access-c6sxc") pod "69358156-ba3f-4e15-9685-424041372e56" (UID: "69358156-ba3f-4e15-9685-424041372e56"). InnerVolumeSpecName "kube-api-access-c6sxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:43:06.581819 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.581781    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6sxc\" (UniqueName: \"kubernetes.io/projected/69358156-ba3f-4e15-9685-424041372e56-kube-api-access-c6sxc\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:43:06.687556 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:06.687528    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69358156-ba3f-4e15-9685-424041372e56" path="/var/lib/kubelet/pods/69358156-ba3f-4e15-9685-424041372e56/volumes"
Jul 10 03:43:07.198934 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:07.198854    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7760-140236786706448"
Jul 10 03:43:07.198934 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:43:07.198906    2551 scope.go:117] "RemoveContainer" containerID="7cce0f897c08e7f638a4f02c38efa4d5b5493be7f171ed0828d403815efaa73f"
Jul 10 03:44:12.399092 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.399051    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:44:12.399639 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.399389    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" podUID="1d0e45b2-749a-47ce-a287-a2665d834fab" containerName="maas-api" containerID="cri-o://9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6" gracePeriod=30
Jul 10 03:44:12.645024 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.644999    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:44:12.744893 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.744794    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wdt2\" (UniqueName: \"kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2\") pod \"1d0e45b2-749a-47ce-a287-a2665d834fab\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") "
Jul 10 03:44:12.744893 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.744837    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca\") pod \"1d0e45b2-749a-47ce-a287-a2665d834fab\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") "
Jul 10 03:44:12.745089 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.744920    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls\") pod \"1d0e45b2-749a-47ce-a287-a2665d834fab\" (UID: \"1d0e45b2-749a-47ce-a287-a2665d834fab\") "
Jul 10 03:44:12.745323 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.745289    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d0e45b2-749a-47ce-a287-a2665d834fab" (UID: "1d0e45b2-749a-47ce-a287-a2665d834fab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 03:44:12.747044 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.747018    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2" (OuterVolumeSpecName: "kube-api-access-7wdt2") pod "1d0e45b2-749a-47ce-a287-a2665d834fab" (UID: "1d0e45b2-749a-47ce-a287-a2665d834fab"). InnerVolumeSpecName "kube-api-access-7wdt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:44:12.747205 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.747014    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d0e45b2-749a-47ce-a287-a2665d834fab" (UID: "1d0e45b2-749a-47ce-a287-a2665d834fab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:44:12.846065 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.846025    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0e45b2-749a-47ce-a287-a2665d834fab-maas-api-tls\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:44:12.846065 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.846059    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdt2\" (UniqueName: \"kubernetes.io/projected/1d0e45b2-749a-47ce-a287-a2665d834fab-kube-api-access-7wdt2\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:44:12.846065 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:12.846068    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e45b2-749a-47ce-a287-a2665d834fab-openshift-service-ca\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:44:13.487309 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.487273    2551 generic.go:356] "Generic (PLEG): container finished" podID="1d0e45b2-749a-47ce-a287-a2665d834fab" containerID="9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6" exitCode=0
Jul 10 03:44:13.487309 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.487312    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" event={"ID":"1d0e45b2-749a-47ce-a287-a2665d834fab","Type":"ContainerDied","Data":"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"}
Jul 10 03:44:13.487757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.487336    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"
Jul 10 03:44:13.487757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.487391    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh" event={"ID":"1d0e45b2-749a-47ce-a287-a2665d834fab","Type":"ContainerDied","Data":"a6f7fa510da25264c768dbbb80bc28d000bb3be9356737f48c8408acf4193d93"}
Jul 10 03:44:13.487757 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.487414    2551 scope.go:117] "RemoveContainer" containerID="9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"
Jul 10 03:44:13.496052 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.496034    2551 scope.go:117] "RemoveContainer" containerID="9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"
Jul 10 03:44:13.496315 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:44:13.496297    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6\": container with ID starting with 9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6 not found: ID does not exist" containerID="9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"
Jul 10 03:44:13.496373 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.496324    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6"} err="failed to get container status \"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6\": rpc error: code = NotFound desc = could not find container \"9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6\": container with ID starting with 9e13962780c876cd10aeea991ccd1799786548cc03d425214d3051a698a9f3e6 not found: ID does not exist"
Jul 10 03:44:13.507437 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.507415    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:44:13.511355 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:13.511334    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7b338-7dfdf84465-vpghh"]
Jul 10 03:44:14.687555 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:44:14.687515    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0e45b2-749a-47ce-a287-a2665d834fab" path="/var/lib/kubelet/pods/1d0e45b2-749a-47ce-a287-a2665d834fab/volumes"
Jul 10 03:45:06.797817 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:06.797702    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:45:06.807051 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:06.807030    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vhtt6_9fa9e957-042c-44df-aa26-00bed3ab749f/ovn-acl-logging/0.log"
Jul 10 03:45:24.827278 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:24.827240    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:45:24.827899 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:24.827477    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" podUID="9de4cbc0-f4cd-4240-bdee-b33865678b40" containerName="maas-api" containerID="cri-o://bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425" gracePeriod=30
Jul 10 03:45:25.071041 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.071016    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:45:25.238383 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.238344    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf4wz\" (UniqueName: \"kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz\") pod \"9de4cbc0-f4cd-4240-bdee-b33865678b40\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") "
Jul 10 03:45:25.238565 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.238421    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls\") pod \"9de4cbc0-f4cd-4240-bdee-b33865678b40\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") "
Jul 10 03:45:25.238565 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.238485    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca\") pod \"9de4cbc0-f4cd-4240-bdee-b33865678b40\" (UID: \"9de4cbc0-f4cd-4240-bdee-b33865678b40\") "
Jul 10 03:45:25.238832 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.238810    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9de4cbc0-f4cd-4240-bdee-b33865678b40" (UID: "9de4cbc0-f4cd-4240-bdee-b33865678b40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 03:45:25.240591 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.240565    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz" (OuterVolumeSpecName: "kube-api-access-tf4wz") pod "9de4cbc0-f4cd-4240-bdee-b33865678b40" (UID: "9de4cbc0-f4cd-4240-bdee-b33865678b40"). InnerVolumeSpecName "kube-api-access-tf4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 03:45:25.240659 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.240589    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9de4cbc0-f4cd-4240-bdee-b33865678b40" (UID: "9de4cbc0-f4cd-4240-bdee-b33865678b40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 03:45:25.339127 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.339070    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de4cbc0-f4cd-4240-bdee-b33865678b40-maas-api-tls\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:45:25.339127 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.339125    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de4cbc0-f4cd-4240-bdee-b33865678b40-openshift-service-ca\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:45:25.339127 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.339135    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf4wz\" (UniqueName: \"kubernetes.io/projected/9de4cbc0-f4cd-4240-bdee-b33865678b40-kube-api-access-tf4wz\") on node \"ip-10-0-136-90.ec2.internal\" DevicePath \"\""
Jul 10 03:45:25.789990 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.789955    2551 generic.go:356] "Generic (PLEG): container finished" podID="9de4cbc0-f4cd-4240-bdee-b33865678b40" containerID="bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425" exitCode=0
Jul 10 03:45:25.790176 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.790009    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"
Jul 10 03:45:25.790176 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.790024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" event={"ID":"9de4cbc0-f4cd-4240-bdee-b33865678b40","Type":"ContainerDied","Data":"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"}
Jul 10 03:45:25.790176 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.790072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf" event={"ID":"9de4cbc0-f4cd-4240-bdee-b33865678b40","Type":"ContainerDied","Data":"60a623a1f9ba214ac643b702acb28e7b07f5b726499125f7cdb1bb213bcc71fe"}
Jul 10 03:45:25.790176 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.790120    2551 scope.go:117] "RemoveContainer" containerID="bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"
Jul 10 03:45:25.798576 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.798555    2551 scope.go:117] "RemoveContainer" containerID="bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"
Jul 10 03:45:25.798842 ip-10-0-136-90 kubenswrapper[2551]: E0710 03:45:25.798820    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425\": container with ID starting with bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425 not found: ID does not exist" containerID="bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"
Jul 10 03:45:25.798887 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.798852    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425"} err="failed to get container status \"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425\": rpc error: code = NotFound desc = could not find container \"bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425\": container with ID starting with bb1a5945bc737da730ded8bbe3577efec0b550dcf175faaa3d1268f123ea1425 not found: ID does not exist"
Jul 10 03:45:25.810549 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.810520    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:45:25.815830 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:25.815806    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-17c732-7b668cc789-rc2sf"]
Jul 10 03:45:26.686803 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:26.686772    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de4cbc0-f4cd-4240-bdee-b33865678b40" path="/var/lib/kubelet/pods/9de4cbc0-f4cd-4240-bdee-b33865678b40/volumes"
Jul 10 03:45:32.068317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:32.068282    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d45f67fff-p8sdh_754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f/authorino/0.log"
Jul 10 03:45:37.556200 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:37.556171    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-h5t8d_8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1/server/0.log"
Jul 10 03:45:37.686827 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:37.686788    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r4m57_63322fa8-8b22-43c8-9938-f94c636f0d4e/manager/0.log"
Jul 10 03:45:37.953054 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:37.953028    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68647b69dd-nsr52_1c04809c-a030-4044-8cb2-db346ed54d73/copy-manifests/0.log"
Jul 10 03:45:37.960901 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:37.960865    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68647b69dd-nsr52_1c04809c-a030-4044-8cb2-db346ed54d73/manager/0.log"
Jul 10 03:45:40.189115 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:40.189063    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d45f67fff-p8sdh_754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f/authorino/0.log"
Jul 10 03:45:40.554989 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:40.554906    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rz4wk_27cf62e1-8629-41e0-a2ba-6909d779857e/kuadrant-console-plugin/0.log"
Jul 10 03:45:41.701061 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:41.701032    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d56d99d47-5trfj_0c083487-9367-4363-a124-a2328ce7de59/kube-auth-proxy/0.log"
Jul 10 03:45:42.711714 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:42.711677    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mqtsb_ed8edd0b-f289-4a47-ba4b-f434fba398c1/main/0.log"
Jul 10 03:45:42.963375 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:42.963287    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5b6rx_06e2630d-9a39-48ce-b567-8e162cc03950/main/0.log"
Jul 10 03:45:43.333802 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:43.333708    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8496c_7fb268d3-b3a0-4b2d-a2c0-1c7dc1f6940e/main/0.log"
Jul 10 03:45:47.988912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.988876    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4zdlx/must-gather-skrfd"]
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989190    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de4cbc0-f4cd-4240-bdee-b33865678b40" containerName="maas-api"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989202    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de4cbc0-f4cd-4240-bdee-b33865678b40" containerName="maas-api"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989214    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69358156-ba3f-4e15-9685-424041372e56" containerName="test-curl-7760-140236786706448"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989220    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="69358156-ba3f-4e15-9685-424041372e56" containerName="test-curl-7760-140236786706448"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989231    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0e45b2-749a-47ce-a287-a2665d834fab" containerName="maas-api"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989237    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0e45b2-749a-47ce-a287-a2665d834fab" containerName="maas-api"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989285    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="69358156-ba3f-4e15-9685-424041372e56" containerName="test-curl-7760-140236786706448"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989294    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0e45b2-749a-47ce-a287-a2665d834fab" containerName="maas-api"
Jul 10 03:45:47.989317 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.989299    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de4cbc0-f4cd-4240-bdee-b33865678b40" containerName="maas-api"
Jul 10 03:45:47.993323 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:47.993300    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.001703 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.001679    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4zdlx\"/\"default-dockercfg-6bsvz\""
Jul 10 03:45:48.004686 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.004668    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zdlx\"/\"openshift-service-ca.crt\""
Jul 10 03:45:48.005353 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.005337    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zdlx\"/\"kube-root-ca.crt\""
Jul 10 03:45:48.036083 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.036048    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zdlx/must-gather-skrfd"]
Jul 10 03:45:48.128055 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.128017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th499\" (UniqueName: \"kubernetes.io/projected/839ed163-cca4-4b04-87e5-dca87c8f2441-kube-api-access-th499\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.128275 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.128127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ed163-cca4-4b04-87e5-dca87c8f2441-must-gather-output\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.228566 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.228520    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ed163-cca4-4b04-87e5-dca87c8f2441-must-gather-output\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.228566 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.228575    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th499\" (UniqueName: \"kubernetes.io/projected/839ed163-cca4-4b04-87e5-dca87c8f2441-kube-api-access-th499\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.228912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.228887    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/839ed163-cca4-4b04-87e5-dca87c8f2441-must-gather-output\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.238597 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.238564    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th499\" (UniqueName: \"kubernetes.io/projected/839ed163-cca4-4b04-87e5-dca87c8f2441-kube-api-access-th499\") pod \"must-gather-skrfd\" (UID: \"839ed163-cca4-4b04-87e5-dca87c8f2441\") " pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.302665 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.302570    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zdlx/must-gather-skrfd"
Jul 10 03:45:48.435144 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.435119    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zdlx/must-gather-skrfd"]
Jul 10 03:45:48.437056 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:45:48.437029    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod839ed163_cca4_4b04_87e5_dca87c8f2441.slice/crio-70380c5bd4e19028842e71fa26aff719205b7fd61c91b3459d80c39981924e46 WatchSource:0}: Error finding container 70380c5bd4e19028842e71fa26aff719205b7fd61c91b3459d80c39981924e46: Status 404 returned error can't find the container with id 70380c5bd4e19028842e71fa26aff719205b7fd61c91b3459d80c39981924e46
Jul 10 03:45:48.886593 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:48.886557    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zdlx/must-gather-skrfd" event={"ID":"839ed163-cca4-4b04-87e5-dca87c8f2441","Type":"ContainerStarted","Data":"70380c5bd4e19028842e71fa26aff719205b7fd61c91b3459d80c39981924e46"}
Jul 10 03:45:49.894451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:49.894413    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zdlx/must-gather-skrfd" event={"ID":"839ed163-cca4-4b04-87e5-dca87c8f2441","Type":"ContainerStarted","Data":"3c4fe81eea8bf35bbbcea9293a67469f0bcb07888984e28466e71a72ecd23583"}
Jul 10 03:45:49.894451 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:49.894459    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zdlx/must-gather-skrfd" event={"ID":"839ed163-cca4-4b04-87e5-dca87c8f2441","Type":"ContainerStarted","Data":"c0c562d4a3fc43dd59475e0bd5c00c3a3dbcdc0f11fbead6380184392db4ec2e"}
Jul 10 03:45:49.912366 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:49.912289    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4zdlx/must-gather-skrfd" podStartSLOduration=2.091623633 podStartE2EDuration="2.912246311s" totalImagesPullingTime="820.622678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:45:49.909891007 +0000 UTC m=+3943.722969106" watchObservedRunningTime="2026-07-10 03:45:49.912246311 +0000 UTC m=+3943.725324405"
Jul 10 03:45:50.842547 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:50.842488    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mg6bl_6063745e-5b20-4c59-b4a0-3b7d34009b94/global-pull-secret-syncer/0.log"
Jul 10 03:45:51.057818 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:51.057785    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l95bb_9af8fba0-1650-4f7a-bc77-9b829352a916/konnectivity-agent/0.log"
Jul 10 03:45:51.082923 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:51.082891    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-90.ec2.internal_0a70112095ce53725b3f8952b09deb60/haproxy/0.log"
Jul 10 03:45:56.516122 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:56.514683    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d45f67fff-p8sdh_754b3f0e-9bf9-4d7d-8ad2-6044ead62f0f/authorino/0.log"
Jul 10 03:45:56.620052 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:56.620016    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rz4wk_27cf62e1-8629-41e0-a2ba-6909d779857e/kuadrant-console-plugin/0.log"
Jul 10 03:45:58.983371 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:58.983335    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b6n4_b2077661-f3aa-45b4-951f-90e85a4300b6/node-exporter/0.log"
Jul 10 03:45:59.011881 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:59.011854    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b6n4_b2077661-f3aa-45b4-951f-90e85a4300b6/kube-rbac-proxy/0.log"
Jul 10 03:45:59.039645 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:45:59.039612    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b6n4_b2077661-f3aa-45b4-951f-90e85a4300b6/init-textfile/0.log"
Jul 10 03:46:01.389013 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.388972    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"]
Jul 10 03:46:01.396803 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.396770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.401997 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.401967    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"]
Jul 10 03:46:01.456756 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.456423    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswx7\" (UniqueName: \"kubernetes.io/projected/99426db1-6c05-43a2-acf2-13d3c8aa189c-kube-api-access-wswx7\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.456756 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.456534    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-podres\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.456756 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.456585    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-lib-modules\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.456756 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.456615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-sys\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.456756 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.456669    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-proc\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.557944 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.557909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswx7\" (UniqueName: \"kubernetes.io/projected/99426db1-6c05-43a2-acf2-13d3c8aa189c-kube-api-access-wswx7\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.557985    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-podres\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558014    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-lib-modules\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558040    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-sys\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-proc\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558404 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558184    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-proc\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558404 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558205    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-podres\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558404 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558224    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-lib-modules\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.558404 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.558238    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99426db1-6c05-43a2-acf2-13d3c8aa189c-sys\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.566172 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.566143    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswx7\" (UniqueName: \"kubernetes.io/projected/99426db1-6c05-43a2-acf2-13d3c8aa189c-kube-api-access-wswx7\") pod \"perf-node-gather-daemonset-66j9h\" (UID: \"99426db1-6c05-43a2-acf2-13d3c8aa189c\") " pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.710783 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.710693    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:01.879912 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.879808    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"]
Jul 10 03:46:01.887845 ip-10-0-136-90 kubenswrapper[2551]: W0710 03:46:01.887785    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99426db1_6c05_43a2_acf2_13d3c8aa189c.slice/crio-968b0362bb2d7b533e006b3baee6678d027c451545574ad3f848c108bfa33f07 WatchSource:0}: Error finding container 968b0362bb2d7b533e006b3baee6678d027c451545574ad3f848c108bfa33f07: Status 404 returned error can't find the container with id 968b0362bb2d7b533e006b3baee6678d027c451545574ad3f848c108bfa33f07
Jul 10 03:46:01.970492 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:01.970346    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h" event={"ID":"99426db1-6c05-43a2-acf2-13d3c8aa189c","Type":"ContainerStarted","Data":"968b0362bb2d7b533e006b3baee6678d027c451545574ad3f848c108bfa33f07"}
Jul 10 03:46:02.976769 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:02.976724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h" event={"ID":"99426db1-6c05-43a2-acf2-13d3c8aa189c","Type":"ContainerStarted","Data":"93f81414bf0a22be38710b666d8e93d394581008bb0c329592c5924878e4f783"}
Jul 10 03:46:02.977191 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:02.976867    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:02.990731 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:02.990674    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h" podStartSLOduration=1.990651083 podStartE2EDuration="1.990651083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 03:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:46:02.989932946 +0000 UTC m=+3956.803011075" watchObservedRunningTime="2026-07-10 03:46:02.990651083 +0000 UTC m=+3956.803729174"
Jul 10 03:46:03.932232 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:03.932202    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dn7g5_1018b000-e21e-4f77-806f-da744fa85e52/dns/0.log"
Jul 10 03:46:03.955016 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:03.954982    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dn7g5_1018b000-e21e-4f77-806f-da744fa85e52/kube-rbac-proxy/0.log"
Jul 10 03:46:03.980010 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:03.979979    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7z8f6_1e33d7e9-3a83-4e47-8f5d-8ca4f4272a26/dns-node-resolver/0.log"
Jul 10 03:46:04.712434 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:04.712400    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgjqj_0a266b18-49bd-41be-9055-71477697c3e1/node-ca/0.log"
Jul 10 03:46:05.821074 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:05.821043    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d56d99d47-5trfj_0c083487-9367-4363-a124-a2328ce7de59/kube-auth-proxy/0.log"
Jul 10 03:46:06.736165 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:06.736132    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6649n_0e8ac0f6-871a-45ea-8330-d1d901a072b9/serve-healthcheck-canary/0.log"
Jul 10 03:46:07.435536 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:07.435503    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qtm94_28f636a3-81c8-4492-a301-98f47fdc30d3/kube-rbac-proxy/0.log"
Jul 10 03:46:07.457753 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:07.457722    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qtm94_28f636a3-81c8-4492-a301-98f47fdc30d3/exporter/0.log"
Jul 10 03:46:07.481264 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:07.481234    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qtm94_28f636a3-81c8-4492-a301-98f47fdc30d3/extractor/0.log"
Jul 10 03:46:08.992735 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:08.992702    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4zdlx/perf-node-gather-daemonset-66j9h"
Jul 10 03:46:09.887132 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:09.887084    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-h5t8d_8ed35d12-3f4d-4c2f-8acd-89dd622ce9e1/server/0.log"
Jul 10 03:46:09.987072 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:09.987037    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r4m57_63322fa8-8b22-43c8-9938-f94c636f0d4e/manager/0.log"
Jul 10 03:46:10.117979 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:10.117944    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68647b69dd-nsr52_1c04809c-a030-4044-8cb2-db346ed54d73/manager/0.log"
Jul 10 03:46:10.139129 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:10.139021    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68647b69dd-nsr52_1c04809c-a030-4044-8cb2-db346ed54d73/copy-manifests/0.log"
Jul 10 03:46:11.667054 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:11.666951    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b767bdcf8-7rngr_2b4d35e1-694c-4e3b-b166-00502d6b2704/manager/0.log"
Jul 10 03:46:12.282061 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:12.282033    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d56d99d47-5trfj_0c083487-9367-4363-a124-a2328ce7de59/kube-auth-proxy/0.log"
Jul 10 03:46:13.816881 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:13.816849    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d56d99d47-5trfj_0c083487-9367-4363-a124-a2328ce7de59/kube-auth-proxy/0.log"
Jul 10 03:46:18.574184 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.574076    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/kube-multus-additional-cni-plugins/0.log"
Jul 10 03:46:18.597285 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.597254    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/egress-router-binary-copy/0.log"
Jul 10 03:46:18.620500 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.620471    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/cni-plugins/0.log"
Jul 10 03:46:18.643024 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.642987    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/bond-cni-plugin/0.log"
Jul 10 03:46:18.668319 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.668285    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/routeoverride-cni/0.log"
Jul 10 03:46:18.690457 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.690432    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/whereabouts-cni-bincopy/0.log"
Jul 10 03:46:18.717164 ip-10-0-136-90 kubenswrapper[2551]: I0710 03:46:18.717128    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2gzv_43824797-43aa-4673-8294-a706183b1fc8/whereabouts-cni/0.log"