Jul 06 15:31:40.529044 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 15:31:40.529053 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 15:31:40.529060 ip-10-0-138-21 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 15:31:40.529285 ip-10-0-138-21 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 15:31:50.763650 ip-10-0-138-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 15:31:50.763666 ip-10-0-138-21 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8d4d877d4900490e9e93b219947df683 --
Jul 06 15:33:53.013956 ip-10-0-138-21 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 15:33:53.409738 ip-10-0-138-21 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:33:53.409738 ip-10-0-138-21 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 15:33:53.409738 ip-10-0-138-21 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:33:53.409738 ip-10-0-138-21 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 15:33:53.409738 ip-10-0-138-21 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:33:53.410719 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.410573    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416861    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416882    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416886    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416889    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416892    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.416886 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416895    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416898    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416903    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416906    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416908    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416911    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416914    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416917    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416920    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416922    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416925    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416928    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416930    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416933    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416935    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416938    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416940    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416943    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416945    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416948    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.417114 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416951    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416953    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416956    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416958    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416961    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416963    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416968    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416975    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416978    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416982    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416986    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416988    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416991    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416994    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.416997    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417000    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417002    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417005    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417007    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.417602 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417010    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417012    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417015    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417017    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417020    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417022    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417028    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417030    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417033    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417035    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417038    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417041    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417044    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417046    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417049    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417051    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417054    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417058    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417060    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417063    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.418119 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417066    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417068    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417071    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417075    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417077    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417080    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417083    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417085    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417088    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417090    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417092    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417095    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417097    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417100    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417102    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417105    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417107    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417110    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417113    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417116    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.418601 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417118    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417121    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417123    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417126    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417129    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417133    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417137    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417140    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417143    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417146    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417149    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417151    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417154    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417157    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417160    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417892    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417900    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417904    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417907    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.419107 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417910    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417913    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417916    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417919    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417922    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417925    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417927    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417930    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417933    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417935    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417938    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417940    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417942    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417946    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417948    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417951    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417953    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417957    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417959    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417962    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.419570 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417964    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417967    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417970    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417972    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417975    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417977    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417980    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417982    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417985    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417987    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417990    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417992    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417995    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.417998    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418002    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418007    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418011    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418015    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418018    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418024    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.420090 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418028    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418032    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418036    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418040    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418042    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418045    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418049    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418051    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418054    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418057    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418059    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418063    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418065    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418067    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418070    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418073    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418075    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418078    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418081    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418083    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.420584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418086    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418088    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418091    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418093    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418096    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418098    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418101    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418103    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418106    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418108    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418111    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418113    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418115    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418118    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418120    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418123    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418125    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418130    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418133    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.421091 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418137    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418141    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418146    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418150    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418154    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418157    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418160    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418163    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418166    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418168    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418171    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418173    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418176    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418178    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418181    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418184    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418568    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418573    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418576    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.421541 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418579    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418581    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418584    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418586    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418588    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418591    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418593    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418596    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418598    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418601    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418603    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418606    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418609    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418612    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418615    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418618    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418621    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418624    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418628    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418632    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.422033 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418635    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418637    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418640    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418642    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418645    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418647    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418651    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418655    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418657    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418676    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418679    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418681    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418684    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418687    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418689    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418692    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418694    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418697    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418700    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.422518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418702    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418706    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418709    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418712    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418715    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418719    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418721    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418725    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418731    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418733    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418736    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418738    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418741    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418743    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418746    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418749    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418751    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418754    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418757    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418759    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.423014 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418762    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418764    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418766    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418769    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418772    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418774    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418776    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418779    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418782    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418784    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418786    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418789    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418791    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418793    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418796    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418798    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418800    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418805    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418807    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418810    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.423527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418817    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418820    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418822    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418825    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418827    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418830    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418833    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418835    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418838    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418841    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418843    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418846    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418848    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418850    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418853    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418855    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.418858    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418929    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418936    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418943    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 15:33:53.424032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418947    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418951    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418954    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418958    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418962    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418965    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418968    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418972    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418975    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418978    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418981    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418984    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418988    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418992    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.418995    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419000    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419002    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419006    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419009    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419013    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419017    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419019    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419022    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 15:33:53.424541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419025    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419028    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419032    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419034    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419037    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419040    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419044    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419047    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419050    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419052    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419055    2567 flags.go:64] FLAG: --enable-server="true"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419057    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419062    2567 flags.go:64] FLAG: --event-burst="100"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419065    2567 flags.go:64] FLAG: --event-qps="50"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419067    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419071    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419073    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419077    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419080    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419083    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419086    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419090    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419092    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419096    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419099    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 15:33:53.425116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419101    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419106    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419108    2567 flags.go:64] FLAG: --feature-gates=""
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419115    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419119    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419122    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419125    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419128    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419131    2567 flags.go:64] FLAG: --help="false"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419134    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419137    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419140    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419143    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419146    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419150    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419153    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419155    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419158    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419161    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419164    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419167    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419170    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419173    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419176    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 15:33:53.425831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419179    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419182    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419184    2567 flags.go:64] FLAG: --lock-file=""
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419187    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419191    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419194    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419199    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419202    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419205    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419209    2567 flags.go:64] FLAG: --logging-format="text"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419212    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419215    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419218    2567 flags.go:64] FLAG: --manifest-url=""
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419220    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419224    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419227    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419231    2567 flags.go:64] FLAG: --max-pods="110"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419234    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419237    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419240    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419243    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419246    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419248    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419251    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419258    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 15:33:53.426423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419265    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419268    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419271    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419273    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419278    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419281    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419284    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419287    2567 flags.go:64] FLAG: --port="10250"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419289    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419292    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-071098053656b00ac"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419295    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419299    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419302    2567 flags.go:64] FLAG: --register-node="true"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419304    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419308    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419311    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419314    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419318    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419322    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419325    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419328    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419330    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419333    2567 flags.go:64] FLAG: --runonce="false"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419336    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419339    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 15:33:53.427067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419342    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419345    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419347    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419350    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419353    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419356    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419359    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419362    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419365    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419367    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419370    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419373    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419375    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419380    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419383    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419385    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419393    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419395    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419399    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419402    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419404    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419407    2567 flags.go:64] FLAG: --v="2"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419411    2567 flags.go:64] FLAG: --version="false"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419421    2567 flags.go:64] FLAG: --vmodule=""
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419426    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 15:33:53.427699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.419429    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419545    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419549    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419552    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419556    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419559    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419561    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419563    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419566    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419569    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419571    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419574    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419577    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419580    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419582    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419585    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419587    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419590    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419592    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419595    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.428357 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419598    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419600    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419602    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419605    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419607    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419610    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419613    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419615    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419618    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419620    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419622    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419627    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419630    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419632    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419635    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419637    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419640    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419642    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419645    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419647    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419649    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.428851 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419652    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419654    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419657    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419672    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419675    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419678    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419680    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419684    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419688    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419691    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419694    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419697    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419700    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419702    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419705    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419708    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419711    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419714    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419717    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.429364 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419719    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419722    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419724    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419728    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419731    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419734    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419737    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419739    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419742    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419744    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419747    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419749    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419752    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419754    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419757    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419759    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419762    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419764    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419767    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.429840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419769    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419772    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419774    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419777    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419779    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419782    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419784    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419786    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419789    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419793    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419796    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419799    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419801    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419804    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419806    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419809    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419813    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419816    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419819    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.430302 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419821    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.419824    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.420494    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.427316    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.427331    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.428000    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.428011    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428114    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428120    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428123    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428126    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428129    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428132    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428134    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.430871 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428137    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428139    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428142    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428145    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428147    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428151    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428153    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428156    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428158    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428160    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428163    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428165    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428168    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428170    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428173    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428175    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428178    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428186    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428189    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428191    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.431220 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428194    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428197    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428200    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428203    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428205    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428207    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428210    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428212    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428215    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428217    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428220    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428222    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428225    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428227    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428230    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428232    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428234    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428237    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428239    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.431769 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428242    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428244    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428246    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428249    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428252    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428254    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428256    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428259    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428261    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428263    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428267    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428276    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428279    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428283    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428287    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428290    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428293    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428296    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428298    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.432248 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428301    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428304    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428306    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428308    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428311    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428313    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428316    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428318    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428320    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428323    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428325    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428327    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428330    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428332    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428335    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428337    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428339    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428342    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428344    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.432784 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428348    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428351    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428353    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428356    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428358    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428361    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428370    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428373    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428376    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428379    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428382    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428384    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428387    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428390    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428392    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.433250 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.428397    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428539    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428544    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428546    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428553    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428557    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428560    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428563    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428565    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428568    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428570    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428573    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428575    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428577    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428580    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428582    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428584    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428587    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428589    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:33:53.433621 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428591    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428594    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428596    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428598    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428601    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428609    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428611    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428614    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428616    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428619    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428621    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428623    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428626    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428628    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428631    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428633    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428635    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428637    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428640    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428642    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:33:53.434160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428644    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428647    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428649    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428651    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428654    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428656    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428679    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428682    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428685    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428688    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428691    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428693    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428695    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428698    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428700    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428703    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428706    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428708    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428716    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428719    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:33:53.434634 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428721    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428724    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428727    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428730    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428734    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428737    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428740    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428742    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428745    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428748    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428751    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428754    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428756    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428759    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428761    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428764    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428766    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428769    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428771    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:33:53.435160 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428773    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428775    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428778    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428781    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428784    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428786    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428788    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428791    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428793    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428795    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428803    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428806    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428815    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428817    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428820    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428822    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428824    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428826    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428829    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428831    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:33:53.435629 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428833    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:53.428836    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.428840    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.428953    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.431099    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.432463    2567 server.go:1019] "Starting client certificate rotation"
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.432624    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.433290    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 15:33:53.436136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.434957    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 15:33:53.441080 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.441057    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 15:33:53.451312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.451295    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 06 15:33:53.453284 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.453269    2567 log.go:25] "Validated CRI v1 image API"
Jul 06 15:33:53.454140 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.454126    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 15:33:53.460823 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.460790    2567 fs.go:135] Filesystem UUIDs: map[65f37098-a725-4e1b-9365-ea7e66d60726:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9c7c3118-171c-44d9-8139-a0f9599e9766:/dev/nvme0n1p4]
Jul 06 15:33:53.460907 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.460817    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 15:33:53.465827 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.465706    2567 manager.go:216] Machine: {Timestamp:2026-07-06 15:33:53.464533892 +0000 UTC m=+0.334581053 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099913 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec235ed32cf6f426d471fe88611dcdc3 SystemUUID:ec235ed3-2cf6-f426-d471-fe88611dcdc3 BootID:8d4d877d-4900-490e-9e93-b219947df683 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:41:d1:bd:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:41:d1:bd:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:16:da:31:02:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 15:33:53.465827 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.465819    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 15:33:53.465999 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.465933    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 15:33:53.467038 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.467009    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 15:33:53.467202 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.467041    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 15:33:53.468677 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.468648    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 15:33:53.468746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.468696    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 15:33:53.468746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.468709    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 15:33:53.468746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.468725    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 15:33:53.469958 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.469945    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 15:33:53.472746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472731    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 15:33:53.472897 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472885    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 15:33:53.472953 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472926    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 15:33:53.472953 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472947    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 15:33:53.473044 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472969    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 15:33:53.473044 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472984    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 06 15:33:53.473044 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.472998    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 15:33:53.473408 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.473394    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 15:33:53.473469 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.473417    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 15:33:53.474883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.474864    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 15:33:53.477169 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.477144    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 15:33:53.478871 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.478856    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 15:33:53.478919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.478889    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 15:33:53.479996 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.479982    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 15:33:53.480041 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480008    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480017    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480054    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480065    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480071    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480079    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 15:33:53.480084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480085    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 15:33:53.480365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480092    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 15:33:53.480365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480098    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 15:33:53.480365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480109    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 15:33:53.480365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.480114    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 15:33:53.481485 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.481473    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 15:33:53.481485 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.481485    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 15:33:53.483580 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.483495    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 15:33:53.483860 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.483833    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 15:33:53.485633 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.485620    2567 server.go:1268] "Started kubelet"
Jul 06 15:33:53.485754 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.485716    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 15:33:53.485837 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.485781    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 15:33:53.485944 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.485857    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 15:33:53.487529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.487509    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 15:33:53.487833 ip-10-0-138-21 systemd[1]: Started Kubernetes Kubelet.
Jul 06 15:33:53.489172 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.489150    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 15:33:53.490416 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.490395    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 15:33:53.494743 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.494726    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 15:33:53.494822 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.494726    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 15:33:53.494877 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.494842    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:53.495540 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495518    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 15:33:53.495540 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495520    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 15:33:53.495860 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495551    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 15:33:53.495860 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495647    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 15:33:53.495860 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495657    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 15:33:53.495860 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495852    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495889    2567 factory.go:153] Registering CRI-O factory
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495907    2567 factory.go:223] Registration of the crio container factory successfully
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495955    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495965    2567 factory.go:55] Registering systemd factory
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495973    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.495994    2567 factory.go:103] Registering Raw factory
Jul 06 15:33:53.496051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.496009    2567 manager.go:1196] Started watching for new ooms in manager
Jul 06 15:33:53.496529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.496515    2567 manager.go:349] Starting recovery of all containers
Jul 06 15:33:53.496588 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.495762    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.ec2.internal.18bfbd39d160f272  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.ec2.internal,UID:ip-10-0-138-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.ec2.internal,},FirstTimestamp:2026-07-06 15:33:53.485582962 +0000 UTC m=+0.355630121,LastTimestamp:2026-07-06 15:33:53.485582962 +0000 UTC m=+0.355630121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.ec2.internal,}"
Jul 06 15:33:53.502182 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.502149    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 15:33:53.502344 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.502319    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 15:33:53.507199 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.507065    2567 manager.go:354] Recovery completed
Jul 06 15:33:53.509184 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.509160    2567 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 06 15:33:53.511115 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.511095    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zxldx"
Jul 06 15:33:53.512492 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.512480    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.514349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514333    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.514422 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514365    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.514422 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514380    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.514983 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514970    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 15:33:53.514983 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514980    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.515123 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.514993    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 15:33:53.515123 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.515011    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 15:33:53.515673 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.515642    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.515724 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.515689    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.515724 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.515700    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.516313 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.516299    2567 policy_none.go:49] "None policy: Start"
Jul 06 15:33:53.516371 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.516320    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 15:33:53.516371 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.516334    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 15:33:53.516700 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.516602    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.ec2.internal.18bfbd39d317dfbc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.ec2.internal,UID:ip-10-0-138-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.ec2.internal,},FirstTimestamp:2026-07-06 15:33:53.514348476 +0000 UTC m=+0.384395689,LastTimestamp:2026-07-06 15:33:53.514348476 +0000 UTC m=+0.384395689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.ec2.internal,}"
Jul 06 15:33:53.517293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.517281    2567 policy_none.go:47] "Start"
Jul 06 15:33:53.521219 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.521153    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zxldx"
Jul 06 15:33:53.558235 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558218    2567 manager.go:337] "Starting Device Plugin manager"
Jul 06 15:33:53.558328 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.558253    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 15:33:53.558328 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558263    2567 server.go:87] "Starting device plugin registration server"
Jul 06 15:33:53.558502 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558491    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 15:33:53.558537 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558509    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 15:33:53.558615 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558600    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 15:33:53.558716 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558700    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 15:33:53.558716 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.558713    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 15:33:53.559135 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.559120    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 15:33:53.559221 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.559165    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:53.656290 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.656250    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 15:33:53.657707 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.657690    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 15:33:53.657783 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.657741    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 15:33:53.657783 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.657768    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 15:33:53.657855 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.657803    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 15:33:53.658645 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.658626    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.659459 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.659440    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.659774 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.659472    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.659774 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.659486    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.659774 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.659512    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.661674 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.661644    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:53.677066 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.677042    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.677187 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.677072    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-21.ec2.internal\": node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:53.715423 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.715397    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:53.758417 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.758377    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"]
Jul 06 15:33:53.758523 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.758460    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.759630 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.759613    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.759741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.759645    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.759741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.759656    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.761019 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761006    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.761129 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.761234 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761148    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.761744 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761729    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.761833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761737    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.761833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761780    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.761833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761793    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.761833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761756    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.762017 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.761837    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.762961 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.762946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.763030 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.762972    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:33:53.763715 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.763699    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:33:53.763782 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.763726    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:33:53.763782 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.763736    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:33:53.776245 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.776227    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.ec2.internal\" not found" node="ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.780156 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.780142    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.ec2.internal\" not found" node="ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.797203 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.797187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.797270 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.797212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.797270 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.797232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57895fd63ca76ed1e35265f2fb017564-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"57895fd63ca76ed1e35265f2fb017564\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.816294 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.816272    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:53.897716 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.897716 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.897859 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57895fd63ca76ed1e35265f2fb017564-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"57895fd63ca76ed1e35265f2fb017564\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.897859 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897784    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57895fd63ca76ed1e35265f2fb017564-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.ec2.internal\" (UID: \"57895fd63ca76ed1e35265f2fb017564\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.897859 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897782    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.897859 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:53.897789    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/930dd80c73085229927a73371c582b4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal\" (UID: \"930dd80c73085229927a73371c582b4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:53.916937 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:53.916883    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.017973 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.017942    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.081591 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.081552    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:54.083878 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.083863    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:54.119143 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.119103    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.220626 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.220591    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.322147 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.322112    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.378235 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.378200    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:54.423526 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.423494    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.432250 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.432231    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 15:33:54.432384 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.432367    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 15:33:54.432440 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.432401    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 15:33:54.490774 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.490700    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:54.522812 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.522777    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 15:28:53 +0000 UTC" deadline="2027-12-01 00:12:00.19928819 +0000 UTC"
Jul 06 15:33:54.522812 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.522808    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12296h38m5.676484029s"
Jul 06 15:33:54.523874 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.523855    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.625201 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:54.625165    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-21.ec2.internal\" not found"
Jul 06 15:33:54.658527 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:54.658495    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930dd80c73085229927a73371c582b4d.slice/crio-5ebd7d1b4dfffbcd2c018345ed1394719971305e6be59dcd34438877990f3693 WatchSource:0}: Error finding container 5ebd7d1b4dfffbcd2c018345ed1394719971305e6be59dcd34438877990f3693: Status 404 returned error can't find the container with id 5ebd7d1b4dfffbcd2c018345ed1394719971305e6be59dcd34438877990f3693
Jul 06 15:33:54.659004 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:54.658982    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57895fd63ca76ed1e35265f2fb017564.slice/crio-8ad998bfcf83fb1f5959573fc53d8463c55c0461672ade87792fb3bccbc6ac52 WatchSource:0}: Error finding container 8ad998bfcf83fb1f5959573fc53d8463c55c0461672ade87792fb3bccbc6ac52: Status 404 returned error can't find the container with id 8ad998bfcf83fb1f5959573fc53d8463c55c0461672ade87792fb3bccbc6ac52
Jul 06 15:33:54.665452 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.665433    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:33:54.718421 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.718390    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:54.799164 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.799071    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:54.813457 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.813433    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 15:33:54.816037 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.814969    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"
Jul 06 15:33:54.824869 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:54.824851    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 15:33:55.473577 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.473544    2567 apiserver.go:52] "Watching apiserver"
Jul 06 15:33:55.480836 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.480811    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 15:33:55.482190 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.481837    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk","openshift-cluster-node-tuning-operator/tuned-7xgt8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal","openshift-multus/multus-additional-cni-plugins-t2b2d","openshift-multus/network-metrics-daemon-btqvz","openshift-dns/node-resolver-5ts55","openshift-image-registry/node-ca-z7gdp","openshift-multus/multus-kplxj","openshift-network-diagnostics/network-check-target-vl8xx","openshift-network-operator/iptables-alerter-bd44r","openshift-ovn-kubernetes/ovnkube-node-488fh","kube-system/global-pull-secret-syncer-d48l4","kube-system/konnectivity-agent-ffjm5","kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal"]
Jul 06 15:33:55.483521 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.483501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.485222 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.485201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.486248 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.486225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.486349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.486328    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.486415 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.486401    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wshd7\""
Jul 06 15:33:55.486465 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.486328    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 15:33:55.486694 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.486657    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.487740 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.487380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 15:33:55.487740 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.487497    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.487740 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.487500    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qp8dh\""
Jul 06 15:33:55.487939 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.487801    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 15:33:55.488261 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.488245    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.489077 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.488880    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrp4m\""
Jul 06 15:33:55.489077 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.488928    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 15:33:55.489228 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.489095    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.489228 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.489193    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:33:55.489346 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.489330    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.489820 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.489783    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 15:33:55.490024 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.490010    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 15:33:55.490177 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.490164    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zgr8v\""
Jul 06 15:33:55.490441 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.490424    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 15:33:55.492140 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.492113    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.493350 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.493335    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:55.493420 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.493403    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:33:55.494960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.494945    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 15:33:55.495086 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.495071    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.495345 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.495328    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.495430 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.495416    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.496094 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.496076    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d2xmv\""
Jul 06 15:33:55.496592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.496577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.497084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.496969    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rvrd2\""
Jul 06 15:33:55.497234 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.497217    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 15:33:55.497747 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.497580    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.497948 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.497932    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.498798 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.498776    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.501520 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.499589    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lf7jx\""
Jul 06 15:33:55.501520 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.500115    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.501520 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.500214    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.502078 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.502063    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 15:33:55.502273 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.502254    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.502339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.502273    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-82k6p\""
Jul 06 15:33:55.502339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.502336    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.502733 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.502716    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.503962 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.503937    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.504034 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.504016    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:33:55.505736 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.505721    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 15:33:55.506637 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.506352    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 15:33:55.506637 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.506368    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 15:33:55.507141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.507226 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-kubernetes\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507226 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9bm\" (UniqueName: \"kubernetes.io/projected/c60ca798-3c9a-48dd-8565-fd717dadb1e1-kube-api-access-cc9bm\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.507226 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507210    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gdz\" (UniqueName: \"kubernetes.io/projected/68ec5342-813b-40d3-8cd6-ed361ac72e0e-kube-api-access-94gdz\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507234    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xdr\" (UniqueName: \"kubernetes.io/projected/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kube-api-access-97xdr\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92943c63-f326-47cb-819d-16d5f00081e3-agent-certs\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507269    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-hostroot\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507290    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92943c63-f326-47cb-819d-16d5f00081e3-konnectivity-ca\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-multus-certs\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-device-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.507374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysconfig\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4fq\" (UniqueName: \"kubernetes.io/projected/d92267dc-44c4-44a4-ab83-c7705592d9b2-kube-api-access-2v4fq\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7fn\" (UniqueName: \"kubernetes.io/projected/f73df3d9-c51a-4e9d-9dd0-1d5129116359-kube-api-access-zc7fn\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507461    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckwgv\""
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-bin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507531    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-kubelet\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507563    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-lib-modules\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-os-release\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507612    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-run\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cnibin\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-os-release\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-tuned\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-socket-dir-parent\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-sys-fs\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.507891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-modprobe-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60ca798-3c9a-48dd-8565-fd717dadb1e1-serviceca\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-tmp\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-hosts-file\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2d57\" (UniqueName: \"kubernetes.io/projected/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-kube-api-access-p2d57\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-system-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.507957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-netns\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-multus\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-daemon-config\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508036    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-host\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-tmp-dir\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60ca798-3c9a-48dd-8565-fd717dadb1e1-host\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-registration-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-sys\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.508585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-system-cni-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508169    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-socket-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-cni-binary-copy\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508198    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vqg\" (UniqueName: \"kubernetes.io/projected/b53f6f2f-4196-4058-8cdd-415efbd9052c-kube-api-access-m6vqg\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508252    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-systemd\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-var-lib-kubelet\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-k8s-cni-cncf-io\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-conf-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-etc-kubernetes\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-conf\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.509380 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.508392    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-cnibin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.513292 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.513271    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 15:33:55.546820 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.546795    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-74hmz"
Jul 06 15:33:55.553635 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.553615    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-74hmz"
Jul 06 15:33:55.596426 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.596398    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 15:33:55.608569 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608536    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60ca798-3c9a-48dd-8565-fd717dadb1e1-host\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.608750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.608750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-registration-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.608750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608608    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60ca798-3c9a-48dd-8565-fd717dadb1e1-host\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.608750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-sys\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608783    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-system-cni-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-systemd-units\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-netns\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608858    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-etc-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-socket-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.608949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-env-overrides\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608968    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-cni-binary-copy\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vqg\" (UniqueName: \"kubernetes.io/projected/b53f6f2f-4196-4058-8cdd-415efbd9052c-kube-api-access-m6vqg\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-systemd\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-var-lib-kubelet\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-ovn\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609114    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-kubelet-config\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.609181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-k8s-cni-cncf-io\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-conf-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-etc-kubernetes\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-conf\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe43f787-6554-46cc-a9e8-d0ac51239066-host-slash\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovn-node-metrics-cert\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-cnibin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-kubernetes\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe43f787-6554-46cc-a9e8-d0ac51239066-iptables-alerter-script\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.609458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlx6\" (UniqueName: \"kubernetes.io/projected/fe43f787-6554-46cc-a9e8-d0ac51239066-kube-api-access-gxlx6\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-var-lib-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9bm\" (UniqueName: \"kubernetes.io/projected/c60ca798-3c9a-48dd-8565-fd717dadb1e1-kube-api-access-cc9bm\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gdz\" (UniqueName: \"kubernetes.io/projected/68ec5342-813b-40d3-8cd6-ed361ac72e0e-kube-api-access-94gdz\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xdr\" (UniqueName: \"kubernetes.io/projected/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kube-api-access-97xdr\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92943c63-f326-47cb-819d-16d5f00081e3-agent-certs\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609630    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-netd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-hostroot\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92943c63-f326-47cb-819d-16d5f00081e3-konnectivity-ca\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-log-socket\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-script-lib\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609828    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-multus-certs\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.609856 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-device-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysconfig\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4fq\" (UniqueName: \"kubernetes.io/projected/d92267dc-44c4-44a4-ab83-c7705592d9b2-kube-api-access-2v4fq\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7fn\" (UniqueName: \"kubernetes.io/projected/f73df3d9-c51a-4e9d-9dd0-1d5129116359-kube-api-access-zc7fn\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.609989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5drh\" (UniqueName: \"kubernetes.io/projected/d1ad6a98-9183-4b21-a096-e4ca9ca40027-kube-api-access-l5drh\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-bin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-kubelet\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-lib-modules\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-os-release\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-systemd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-node-log\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-run\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cnibin\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610305    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-os-release\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-tuned\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610375 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-socket-dir-parent\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-sys-fs\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610415    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-modprobe-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-slash\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-dbus\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60ca798-3c9a-48dd-8565-fd717dadb1e1-serviceca\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-tmp\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-hosts-file\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610607    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2d57\" (UniqueName: \"kubernetes.io/projected/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-kube-api-access-p2d57\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-config\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-system-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-netns\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-multus\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-daemon-config\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610810    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-host\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.610960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-tmp-dir\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-kubelet\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-bin\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.610916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.608741    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.611051    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:33:56.111020156 +0000 UTC m=+2.981067326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.611239    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-system-cni-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608746    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-sys\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.608745    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-registration-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.611568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.611415    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-socket-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.612010 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.611985    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-cni-binary-copy\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.612371 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612350    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.612477 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612414    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-systemd\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.612477 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612463    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-var-lib-kubelet\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.612563 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612529    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-k8s-cni-cncf-io\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.612603 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612574    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-conf-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.612654 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612614    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-etc-kubernetes\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.612771 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.612756    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-conf\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.613052 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613030    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-kubelet\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.613140 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613127    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-cnibin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.613191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613171    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.613246 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613216    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-kubernetes\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.613291 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613242    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.613386 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613371    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-lib-modules\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.613461 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613440    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-os-release\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.613896 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613874    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.613996 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613965    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-run\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.613996 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.613980    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 15:33:55.614089 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614010    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cnibin\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.614089 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614069    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-os-release\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.614474 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614454    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-hostroot\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.614606 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614522    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysctl-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.614752 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614724    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-socket-dir-parent\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.614852 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614764    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-sys-fs\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.614852 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614847    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-modprobe-d\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.614976 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614885    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-host\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.614976 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614946    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92943c63-f326-47cb-819d-16d5f00081e3-konnectivity-ca\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.614976 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614957    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-device-dir\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.615117 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.614989    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-sysconfig\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.615117 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615000    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-multus\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615204 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615119    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-hosts-file\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.615386 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615332    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615491 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615438    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60ca798-3c9a-48dd-8565-fd717dadb1e1-serviceca\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.615491 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615482    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-tmp-dir\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.615641 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615498    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-var-lib-cni-bin\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615641 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615601    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-netns\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615641 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615605    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-system-cni-dir\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615641 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615620    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53f6f2f-4196-4058-8cdd-415efbd9052c-host-run-multus-certs\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.615874 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615746    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73df3d9-c51a-4e9d-9dd0-1d5129116359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.615925 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615889    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73df3d9-c51a-4e9d-9dd0-1d5129116359-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.615997 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.615972    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53f6f2f-4196-4058-8cdd-415efbd9052c-multus-daemon-config\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.617837 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.617775    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92943c63-f326-47cb-819d-16d5f00081e3-agent-certs\") pod \"konnectivity-agent-ffjm5\" (UID: \"92943c63-f326-47cb-819d-16d5f00081e3\") " pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.617837 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.617831    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-tmp\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.617963 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.617782    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d92267dc-44c4-44a4-ab83-c7705592d9b2-etc-tuned\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.621686 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.621649    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vqg\" (UniqueName: \"kubernetes.io/projected/b53f6f2f-4196-4058-8cdd-415efbd9052c-kube-api-access-m6vqg\") pod \"multus-kplxj\" (UID: \"b53f6f2f-4196-4058-8cdd-415efbd9052c\") " pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.622048 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.622033    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:33:55.622111 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.622051    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:33:55.622111 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.622061    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:55.622190 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.622112    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:33:56.122099389 +0000 UTC m=+2.992146549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:55.625337 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.625160    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4fq\" (UniqueName: \"kubernetes.io/projected/d92267dc-44c4-44a4-ab83-c7705592d9b2-kube-api-access-2v4fq\") pod \"tuned-7xgt8\" (UID: \"d92267dc-44c4-44a4-ab83-c7705592d9b2\") " pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.625555 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.625533    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xdr\" (UniqueName: \"kubernetes.io/projected/55d8f6b2-1a1d-4e96-abc0-bffc3b51d598-kube-api-access-97xdr\") pod \"aws-ebs-csi-driver-node-hb4jk\" (UID: \"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.625631 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.625608    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gdz\" (UniqueName: \"kubernetes.io/projected/68ec5342-813b-40d3-8cd6-ed361ac72e0e-kube-api-access-94gdz\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:55.626123 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.626098    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9bm\" (UniqueName: \"kubernetes.io/projected/c60ca798-3c9a-48dd-8565-fd717dadb1e1-kube-api-access-cc9bm\") pod \"node-ca-z7gdp\" (UID: \"c60ca798-3c9a-48dd-8565-fd717dadb1e1\") " pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.626945 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.626921    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2d57\" (UniqueName: \"kubernetes.io/projected/2cb8845b-4fb5-40c1-b7ae-399f78bf1d94-kube-api-access-p2d57\") pod \"node-resolver-5ts55\" (UID: \"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94\") " pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.628138 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.628119    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7fn\" (UniqueName: \"kubernetes.io/projected/f73df3d9-c51a-4e9d-9dd0-1d5129116359-kube-api-access-zc7fn\") pod \"multus-additional-cni-plugins-t2b2d\" (UID: \"f73df3d9-c51a-4e9d-9dd0-1d5129116359\") " pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.662099 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.662052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"930dd80c73085229927a73371c582b4d","Type":"ContainerStarted","Data":"5ebd7d1b4dfffbcd2c018345ed1394719971305e6be59dcd34438877990f3693"}
Jul 06 15:33:55.663128 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.663101    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"57895fd63ca76ed1e35265f2fb017564","Type":"ContainerStarted","Data":"8ad998bfcf83fb1f5959573fc53d8463c55c0461672ade87792fb3bccbc6ac52"}
Jul 06 15:33:55.711191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711159    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-slash\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-dbus\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-config\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-kubelet\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711255    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-slash\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-bin\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711322    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-bin\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711321    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711369    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-systemd-units\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711406    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-systemd-units\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-netns\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711436    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-etc-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711440    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-dbus\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711445    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-kubelet\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711495    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-env-overrides\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711548    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-netns\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711586    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-etc-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-ovn\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-kubelet-config\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe43f787-6554-46cc-a9e8-d0ac51239066-host-slash\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovn-node-metrics-cert\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe43f787-6554-46cc-a9e8-d0ac51239066-iptables-alerter-script\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711781    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb24f93f-b357-4b8d-a604-6d89306a5dd6-kubelet-config\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlx6\" (UniqueName: \"kubernetes.io/projected/fe43f787-6554-46cc-a9e8-d0ac51239066-kube-api-access-gxlx6\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-var-lib-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.711957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-netd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711887    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-log-socket\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-script-lib\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5drh\" (UniqueName: \"kubernetes.io/projected/d1ad6a98-9183-4b21-a096-e4ca9ca40027-kube-api-access-l5drh\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-systemd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.711995    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-env-overrides\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-node-log\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712048    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe43f787-6554-46cc-a9e8-d0ac51239066-host-slash\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712049    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-log-socket\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712074    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-config\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712085    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-node-log\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712079    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-ovn\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712107    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-var-lib-openvswitch\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712131    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-run-ovn-kubernetes\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.712180    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712188    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-run-systemd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.712739 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:55.712232    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:56.212215436 +0000 UTC m=+3.082262582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:55.713370 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712394    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1ad6a98-9183-4b21-a096-e4ca9ca40027-host-cni-netd\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.713370 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712629    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe43f787-6554-46cc-a9e8-d0ac51239066-iptables-alerter-script\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.713370 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.712745    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovnkube-script-lib\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.714808 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.714789    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1ad6a98-9183-4b21-a096-e4ca9ca40027-ovn-node-metrics-cert\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.727478 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.727412    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5drh\" (UniqueName: \"kubernetes.io/projected/d1ad6a98-9183-4b21-a096-e4ca9ca40027-kube-api-access-l5drh\") pod \"ovnkube-node-488fh\" (UID: \"d1ad6a98-9183-4b21-a096-e4ca9ca40027\") " pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.728288 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.728270    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlx6\" (UniqueName: \"kubernetes.io/projected/fe43f787-6554-46cc-a9e8-d0ac51239066-kube-api-access-gxlx6\") pod \"iptables-alerter-bd44r\" (UID: \"fe43f787-6554-46cc-a9e8-d0ac51239066\") " pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:55.794519 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.794486    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7gdp"
Jul 06 15:33:55.801160 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.801138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:33:55.811785 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.811763    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kplxj"
Jul 06 15:33:55.820551 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.820531    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:33:55.824179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.824164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2b2d"
Jul 06 15:33:55.829597 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.829579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5ts55"
Jul 06 15:33:55.840184 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.840159    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk"
Jul 06 15:33:55.846750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.846733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xgt8"
Jul 06 15:33:55.847708 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.847690    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:55.852882 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:55.852864    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bd44r"
Jul 06 15:33:56.114806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.114710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:56.114948 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.114880    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:56.114948 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.114943    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.114928326 +0000 UTC m=+3.984975472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:56.201419 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.201390    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53f6f2f_4196_4058_8cdd_415efbd9052c.slice/crio-0ea359663186561ca0d2d0108bbbde93807a2adc83a2c7c4d209e68f74fe64c5 WatchSource:0}: Error finding container 0ea359663186561ca0d2d0108bbbde93807a2adc83a2c7c4d209e68f74fe64c5: Status 404 returned error can't find the container with id 0ea359663186561ca0d2d0108bbbde93807a2adc83a2c7c4d209e68f74fe64c5
Jul 06 15:33:56.202612 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.202575    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60ca798_3c9a_48dd_8565_fd717dadb1e1.slice/crio-107829ba8fd1c0d0dd0c52d24e49c015b011db3d62f774601fa349b4fea88423 WatchSource:0}: Error finding container 107829ba8fd1c0d0dd0c52d24e49c015b011db3d62f774601fa349b4fea88423: Status 404 returned error can't find the container with id 107829ba8fd1c0d0dd0c52d24e49c015b011db3d62f774601fa349b4fea88423
Jul 06 15:33:56.203620 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.203531    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb8845b_4fb5_40c1_b7ae_399f78bf1d94.slice/crio-bd749e0a9416d228805044c07ac43a5dabf0a8958738310bd917eaa8d0e81b5b WatchSource:0}: Error finding container bd749e0a9416d228805044c07ac43a5dabf0a8958738310bd917eaa8d0e81b5b: Status 404 returned error can't find the container with id bd749e0a9416d228805044c07ac43a5dabf0a8958738310bd917eaa8d0e81b5b
Jul 06 15:33:56.204503 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.204471    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe43f787_6554_46cc_a9e8_d0ac51239066.slice/crio-f476e98eaad7d386658075c22f7b3fcfc94e25126ebead501caa30e42525a98e WatchSource:0}: Error finding container f476e98eaad7d386658075c22f7b3fcfc94e25126ebead501caa30e42525a98e: Status 404 returned error can't find the container with id f476e98eaad7d386658075c22f7b3fcfc94e25126ebead501caa30e42525a98e
Jul 06 15:33:56.205061 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.205040    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ad6a98_9183_4b21_a096_e4ca9ca40027.slice/crio-a2d89f46739fc0e7aea6476949ec6ad040ca5cbbefd9f18fab9ce0e90239ba15 WatchSource:0}: Error finding container a2d89f46739fc0e7aea6476949ec6ad040ca5cbbefd9f18fab9ce0e90239ba15: Status 404 returned error can't find the container with id a2d89f46739fc0e7aea6476949ec6ad040ca5cbbefd9f18fab9ce0e90239ba15
Jul 06 15:33:56.210019 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.209994    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d8f6b2_1a1d_4e96_abc0_bffc3b51d598.slice/crio-17a5b6a073e4a5daeebfa80aad91c2ff69021032861e7a16c9d0a44a74dbe91d WatchSource:0}: Error finding container 17a5b6a073e4a5daeebfa80aad91c2ff69021032861e7a16c9d0a44a74dbe91d: Status 404 returned error can't find the container with id 17a5b6a073e4a5daeebfa80aad91c2ff69021032861e7a16c9d0a44a74dbe91d
Jul 06 15:33:56.211285 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.210607    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92943c63_f326_47cb_819d_16d5f00081e3.slice/crio-c0c79fb907a65b8a6a8daf4b2973102d1409a2ee2f849e0f4df4eb103735ee93 WatchSource:0}: Error finding container c0c79fb907a65b8a6a8daf4b2973102d1409a2ee2f849e0f4df4eb103735ee93: Status 404 returned error can't find the container with id c0c79fb907a65b8a6a8daf4b2973102d1409a2ee2f849e0f4df4eb103735ee93
Jul 06 15:33:56.211546 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.211504    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73df3d9_c51a_4e9d_9dd0_1d5129116359.slice/crio-47e801146c29f7320983b25a23fea1daf9ee1808ddb30aea08b6731c1013ccc3 WatchSource:0}: Error finding container 47e801146c29f7320983b25a23fea1daf9ee1808ddb30aea08b6731c1013ccc3: Status 404 returned error can't find the container with id 47e801146c29f7320983b25a23fea1daf9ee1808ddb30aea08b6731c1013ccc3
Jul 06 15:33:56.212521 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:33:56.212506    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92267dc_44c4_44a4_ab83_c7705592d9b2.slice/crio-a63700bfb67ccd1e78732535246ed8e2291872356883ff8bc761ac4208c91c62 WatchSource:0}: Error finding container a63700bfb67ccd1e78732535246ed8e2291872356883ff8bc761ac4208c91c62: Status 404 returned error can't find the container with id a63700bfb67ccd1e78732535246ed8e2291872356883ff8bc761ac4208c91c62
Jul 06 15:33:56.215843 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.215816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:56.215923 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.215885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:56.216052 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.215995    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:56.216052 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.216043    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.216025394 +0000 UTC m=+4.086072563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:56.216159 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.216125    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:33:56.216159 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.216139    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:33:56.216159 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.216151    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:56.216299 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.216186    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.216175768 +0000 UTC m=+4.086222928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:56.558738 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.556498    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:28:55 +0000 UTC" deadline="2028-04-17 11:13:58.145394758 +0000 UTC"
Jul 06 15:33:56.558738 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.556791    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15619h40m1.588611958s"
Jul 06 15:33:56.659105 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.658313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:56.659105 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.658445    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:33:56.659105 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.658943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:56.659105 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:56.659045    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:33:56.669833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.669201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" event={"ID":"57895fd63ca76ed1e35265f2fb017564","Type":"ContainerStarted","Data":"81aa34f6c2e40571a90babf18304a318847cbb99c886a2355a7cb5eb13239772"}
Jul 06 15:33:56.671787 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.671761    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerStarted","Data":"47e801146c29f7320983b25a23fea1daf9ee1808ddb30aea08b6731c1013ccc3"}
Jul 06 15:33:56.674925 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.674895    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xgt8" event={"ID":"d92267dc-44c4-44a4-ab83-c7705592d9b2","Type":"ContainerStarted","Data":"a63700bfb67ccd1e78732535246ed8e2291872356883ff8bc761ac4208c91c62"}
Jul 06 15:33:56.677372 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.677300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ffjm5" event={"ID":"92943c63-f326-47cb-819d-16d5f00081e3","Type":"ContainerStarted","Data":"c0c79fb907a65b8a6a8daf4b2973102d1409a2ee2f849e0f4df4eb103735ee93"}
Jul 06 15:33:56.680071 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.680048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"a2d89f46739fc0e7aea6476949ec6ad040ca5cbbefd9f18fab9ce0e90239ba15"}
Jul 06 15:33:56.681568 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.681541    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7gdp" event={"ID":"c60ca798-3c9a-48dd-8565-fd717dadb1e1","Type":"ContainerStarted","Data":"107829ba8fd1c0d0dd0c52d24e49c015b011db3d62f774601fa349b4fea88423"}
Jul 06 15:33:56.683241 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.683219    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kplxj" event={"ID":"b53f6f2f-4196-4058-8cdd-415efbd9052c","Type":"ContainerStarted","Data":"0ea359663186561ca0d2d0108bbbde93807a2adc83a2c7c4d209e68f74fe64c5"}
Jul 06 15:33:56.686473 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.686451    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk" event={"ID":"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598","Type":"ContainerStarted","Data":"17a5b6a073e4a5daeebfa80aad91c2ff69021032861e7a16c9d0a44a74dbe91d"}
Jul 06 15:33:56.688354 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.688332    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bd44r" event={"ID":"fe43f787-6554-46cc-a9e8-d0ac51239066","Type":"ContainerStarted","Data":"f476e98eaad7d386658075c22f7b3fcfc94e25126ebead501caa30e42525a98e"}
Jul 06 15:33:56.690632 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.690587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5ts55" event={"ID":"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94","Type":"ContainerStarted","Data":"bd749e0a9416d228805044c07ac43a5dabf0a8958738310bd917eaa8d0e81b5b"}
Jul 06 15:33:56.892422 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:56.892311    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:33:57.120967 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.120928    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:57.121169 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.121085    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:57.121169 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.121153    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:33:59.121133999 +0000 UTC m=+5.991181163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.221684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.221761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.221886    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.221951    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:59.221933953 +0000 UTC m=+6.091981103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.222363    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.222381    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.222393    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:57.222474 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.222439    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:33:59.22242463 +0000 UTC m=+6.092471781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:57.557505 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.557421    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:28:55 +0000 UTC" deadline="2027-12-12 22:16:38.90423763 +0000 UTC"
Jul 06 15:33:57.557505 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.557459    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12582h42m41.346782543s"
Jul 06 15:33:57.659322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:57.659290    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:57.659924 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:57.659416    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:33:58.658807 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:58.658132    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:58.658807 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:58.658269    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:33:58.658807 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:58.658644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:58.658807 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:58.658756    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:33:58.706627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:58.706591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"930dd80c73085229927a73371c582b4d","Type":"ContainerStarted","Data":"c5b407caa2011e36e2b4a54d812d33248c6300f7133d2370d839801c78918165"}
Jul 06 15:33:58.732027 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:58.731970    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.ec2.internal" podStartSLOduration=4.731948132 podStartE2EDuration="4.731948132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:56.686321314 +0000 UTC m=+3.556368482" watchObservedRunningTime="2026-07-06 15:33:58.731948132 +0000 UTC m=+5.601995302"
Jul 06 15:33:59.136267 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:59.136227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:33:59.136489 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.136450    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:59.136561 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.136513    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:34:03.136494551 +0000 UTC m=+10.006541709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:33:59.237426 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:59.237382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:59.237617 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:59.237477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:33:59.237702 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.237616    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:59.237702 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.237694    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:03.237658876 +0000 UTC m=+10.107706042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:33:59.238150 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.238131    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:33:59.238244 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.238155    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:33:59.238244 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.238168    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:59.238244 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.238213    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:34:03.238197837 +0000 UTC m=+10.108245002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:33:59.659916 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:33:59.659883    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:33:59.660148 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:33:59.660067    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:00.658576 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:00.658538    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:00.659066 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:00.658703    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:00.659066 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:00.658782    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:00.659066 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:00.658902    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:01.658567 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:01.658534    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:01.658746 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:01.658678    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:01.715582 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:01.715542    2567 generic.go:356] "Generic (PLEG): container finished" podID="930dd80c73085229927a73371c582b4d" containerID="c5b407caa2011e36e2b4a54d812d33248c6300f7133d2370d839801c78918165" exitCode=0
Jul 06 15:34:01.715776 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:01.715595    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"930dd80c73085229927a73371c582b4d","Type":"ContainerDied","Data":"c5b407caa2011e36e2b4a54d812d33248c6300f7133d2370d839801c78918165"}
Jul 06 15:34:02.658829 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:02.658201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:02.658829 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:02.658322    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:02.658829 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:02.658684    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:02.658829 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:02.658789    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:03.168559 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:03.168522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:03.168771 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.168654    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:03.168771 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.168734    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:34:11.168715047 +0000 UTC m=+18.038762206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:03.269803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:03.269884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270019    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270033    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270042    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270083    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:34:11.270070533 +0000 UTC m=+18.140117681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270401    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:03.270467 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.270441    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:11.270430815 +0000 UTC m=+18.140477965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:03.658963 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:03.658926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:03.659420 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:03.659041    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:04.658609 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:04.658573    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:04.658828 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:04.658783    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:04.659241 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:04.659222    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:04.659607 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:04.659330    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:05.659007 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:05.658970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:05.659179 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:05.659092    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:06.657991 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:06.657957    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:06.658385 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:06.657976    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:06.658385 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:06.658110    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:06.658385 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:06.658128    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:07.659010 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:07.658977    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:07.659370 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:07.659093    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:08.657981 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:08.657953    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:08.658142 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:08.658056    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:08.658142 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:08.658112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:08.658223 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:08.658204    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:09.658271 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:09.658233    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:09.658715 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:09.658364    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:10.658941 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:10.658897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:10.659440 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:10.659040    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:10.659440 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:10.659105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:10.659440 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:10.659229    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:11.228762 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:11.228730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:11.228974 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.228875    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:11.228974 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.228939    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:34:27.228923758 +0000 UTC m=+34.098970920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:11.329281 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:11.329196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:11.329281 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:11.329268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329371    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329388    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329403    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329416    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329451    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:27.329429601 +0000 UTC m=+34.199476766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:11.329490 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.329474    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:34:27.329458781 +0000 UTC m=+34.199505930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:11.658522 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:11.658443    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:11.658690 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:11.658579    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:12.658295 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:12.658259    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:12.658757 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:12.658260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:12.658757 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:12.658406    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:12.658757 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:12.658453    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:13.659768 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:13.659246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:13.659768 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:13.659374    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:14.658365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:14.658335    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:14.658365 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:14.658343    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:14.658598 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:14.658452    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:14.658598 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:14.658575    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:15.659594 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.659100    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:15.659594 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:15.659454    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:15.743782 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.743550    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kplxj" event={"ID":"b53f6f2f-4196-4058-8cdd-415efbd9052c","Type":"ContainerStarted","Data":"4ee22bc4447d6e8fec989cbdd09cbac8759b1f03f2bd52c29d8d34617ac254f2"}
Jul 06 15:34:15.746115 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.746091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" event={"ID":"930dd80c73085229927a73371c582b4d","Type":"ContainerStarted","Data":"7bde49a30aea81dd7d4ded979e67537508c49c648801432fe024694926b2ce59"}
Jul 06 15:34:15.747657 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.747635    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk" event={"ID":"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598","Type":"ContainerStarted","Data":"098e8370050b2fe0504978abf5aab782897465da372a8e074db2e862bea756e9"}
Jul 06 15:34:15.764179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.764083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5ts55" event={"ID":"2cb8845b-4fb5-40c1-b7ae-399f78bf1d94","Type":"ContainerStarted","Data":"2f6993fd4206725fc3b026c895ec1981681087fd1673386f937a4d882bfa3247"}
Jul 06 15:34:15.765824 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.765797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerStarted","Data":"84997fe789cd0eda41dd36bc02502f59998976a52666e57bc0f37f082a0ad931"}
Jul 06 15:34:15.767403 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.767379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xgt8" event={"ID":"d92267dc-44c4-44a4-ab83-c7705592d9b2","Type":"ContainerStarted","Data":"f9f3466f6c3eceeffe77908ec8727bd2f7795d68c25ec89b2c84272cace5b36c"}
Jul 06 15:34:15.769011 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.768988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ffjm5" event={"ID":"92943c63-f326-47cb-819d-16d5f00081e3","Type":"ContainerStarted","Data":"13f3b45247ecb3a28125aca9139044d8da91f7c894b818096ed9b7f08df920f4"}
Jul 06 15:34:15.770393 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.770375    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"05e4368c5eb42bf517849505bec517c4e1d9b7067f4757e28836219fe43c3c95"}
Jul 06 15:34:15.770453 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.770428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"06cd00fb817bf53dda57eea657a016a942b8a15074190dec6aa798c0738ea1a4"}
Jul 06 15:34:15.773301 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.773273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7gdp" event={"ID":"c60ca798-3c9a-48dd-8565-fd717dadb1e1","Type":"ContainerStarted","Data":"0e36c654042ec31b1b5d794db22c5e9e3f5da4b136dd23248dcebe037c85ab2d"}
Jul 06 15:34:15.776588 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.776539    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kplxj" podStartSLOduration=3.496800842 podStartE2EDuration="22.776523191s" totalImagesPullingTime="19.279722349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.776457721 +0000 UTC m=+22.646504901" watchObservedRunningTime="2026-07-06 15:34:15.776523191 +0000 UTC m=+22.646570360"
Jul 06 15:34:15.821961 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.821922    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xgt8" podStartSLOduration=3.605319265 podStartE2EDuration="22.821906479s" totalImagesPullingTime="19.216587214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.821500785 +0000 UTC m=+22.691547953" watchObservedRunningTime="2026-07-06 15:34:15.821906479 +0000 UTC m=+22.691953646"
Jul 06 15:34:15.822172 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.822145    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5ts55" podStartSLOduration=3.600047958 podStartE2EDuration="22.822140472s" totalImagesPullingTime="19.222092514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.797079785 +0000 UTC m=+22.667126954" watchObservedRunningTime="2026-07-06 15:34:15.822140472 +0000 UTC m=+22.692187639"
Jul 06 15:34:15.841257 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.841203    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.ec2.internal" podStartSLOduration=21.841189732 podStartE2EDuration="21.841189732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.841013975 +0000 UTC m=+22.711061143" watchObservedRunningTime="2026-07-06 15:34:15.841189732 +0000 UTC m=+22.711236901"
Jul 06 15:34:15.862650 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.862609    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z7gdp" podStartSLOduration=3.6413287949999997 podStartE2EDuration="22.862593799s" totalImagesPullingTime="19.221265004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.86204654 +0000 UTC m=+22.732093707" watchObservedRunningTime="2026-07-06 15:34:15.862593799 +0000 UTC m=+22.732641045"
Jul 06 15:34:15.890911 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:15.890873    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ffjm5" podStartSLOduration=11.879264506 podStartE2EDuration="22.890857612s" totalImagesPullingTime="11.011593106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:15.890408968 +0000 UTC m=+22.760456138" watchObservedRunningTime="2026-07-06 15:34:15.890857612 +0000 UTC m=+22.760904780"
Jul 06 15:34:16.658729 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.658695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:16.658913 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:16.658805    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:16.658913 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.658857    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:16.658993 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:16.658944    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:16.776909 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.776832    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="84997fe789cd0eda41dd36bc02502f59998976a52666e57bc0f37f082a0ad931" exitCode=0
Jul 06 15:34:16.777569 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.776914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"84997fe789cd0eda41dd36bc02502f59998976a52666e57bc0f37f082a0ad931"}
Jul 06 15:34:16.780247 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.780216    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"2372201060e131f73ab3310c54e738d01f647542aaffbb63d50bf1a4a75266e0"}
Jul 06 15:34:16.780308 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.780281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"613f54d85157ed7a60e45688955483777f25819b18c00f52f1acc4529a811830"}
Jul 06 15:34:16.780350 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.780310    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"eaf3bb52346d422d8527bf263d8bf75d2ef64457612d1d9366befc801c558d99"}
Jul 06 15:34:16.780350 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:16.780339    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"f06e17ed2602d6ed73b6a7ccede2f64536acdc7a6840682e11d044918c416e61"}
Jul 06 15:34:17.013536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.013511    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 15:34:17.570519 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.570406    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T15:34:17.013530491Z","UUID":"77723860-202c-4446-a8e6-da95698baf80","Handler":null,"Name":"","Endpoint":""}
Jul 06 15:34:17.572120 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.572095    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 15:34:17.572293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.572132    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 15:34:17.605444 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.605414    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:34:17.606035 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.606017    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:34:17.659008 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.658979    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:17.659176 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:17.659102    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:17.784606 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.784565    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk" event={"ID":"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598","Type":"ContainerStarted","Data":"73227e51d6f013bb6128675478ba045ec01e3bcab607178ab3c852d8cb8b6990"}
Jul 06 15:34:17.787895 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.787834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bd44r" event={"ID":"fe43f787-6554-46cc-a9e8-d0ac51239066","Type":"ContainerStarted","Data":"bdc1718b84b0b43fccb42e1dc0ce0723f2cebff2a76e10846c4a592e30d31931"}
Jul 06 15:34:17.788166 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.788148    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:34:17.788582 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.788564    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ffjm5"
Jul 06 15:34:17.803207 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:17.803162    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bd44r" podStartSLOduration=5.585171692 podStartE2EDuration="24.803149012s" totalImagesPullingTime="19.21797732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:17.802979154 +0000 UTC m=+24.673026322" watchObservedRunningTime="2026-07-06 15:34:17.803149012 +0000 UTC m=+24.673196180"
Jul 06 15:34:18.658708 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:18.658611    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:18.658708 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:18.658624    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:18.659002 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:18.658765    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:18.659002 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:18.658874    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:18.794492 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:18.794454    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"7d60b388826ad746a9806136abc6f66fcc9421e559f563cf598b86ad6253da4d"}
Jul 06 15:34:18.796809 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:18.796780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk" event={"ID":"55d8f6b2-1a1d-4e96-abc0-bffc3b51d598","Type":"ContainerStarted","Data":"d328476e802272c7a6370a5a9d631b7883eb4db76a59dd3042718e4413e2efb5"}
Jul 06 15:34:18.815723 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:18.815654    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb4jk" podStartSLOduration=8.824308867 podStartE2EDuration="25.815639265s" totalImagesPullingTime="16.991330398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:18.81514763 +0000 UTC m=+25.685194799" watchObservedRunningTime="2026-07-06 15:34:18.815639265 +0000 UTC m=+25.685686432"
Jul 06 15:34:19.658263 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:19.658228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:19.658444 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:19.658329    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:20.658653 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.658424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:20.659331 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.658446    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:20.659331 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:20.658739    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:20.659331 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:20.658751    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:20.804421 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.804384    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" event={"ID":"d1ad6a98-9183-4b21-a096-e4ca9ca40027","Type":"ContainerStarted","Data":"e4e825eb33007a74785c3b132750021df09617a1a61dfd012f0ed17946ec267f"}
Jul 06 15:34:20.804692 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.804651    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:20.819969 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.819942    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:20.836524 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:20.836487    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-488fh" podStartSLOduration=8.529246998 podStartE2EDuration="27.836473271s" totalImagesPullingTime="19.307226273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:20.836079249 +0000 UTC m=+27.706126418" watchObservedRunningTime="2026-07-06 15:34:20.836473271 +0000 UTC m=+27.706520439"
Jul 06 15:34:21.658795 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:21.658715    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:21.659336 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:21.658860    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:21.808428 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:21.808402    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 15:34:21.808945 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:21.808918    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:21.826364 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:21.826340    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:22.658525 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.658339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:22.658706 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.658380    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:22.658788 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:22.658624    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:22.658859 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:22.658773    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:22.810439 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.810412    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 15:34:22.904321 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.904289    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-btqvz"]
Jul 06 15:34:22.904457 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.904394    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:22.904501 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:22.904483    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:22.905229 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.904889    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d48l4"]
Jul 06 15:34:22.905229 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.904959    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:22.905229 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:22.905049    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:22.910265 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.910191    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vl8xx"]
Jul 06 15:34:22.910362 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:22.910289    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:22.910409 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:22.910382    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:23.763597 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:23.763562    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:23.813921 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:23.813891    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="c48b0740162421feb92aef958257dc23218371b1ccadbee2e8407217e5c2bebe" exitCode=0
Jul 06 15:34:23.814055 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:23.813971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"c48b0740162421feb92aef958257dc23218371b1ccadbee2e8407217e5c2bebe"}
Jul 06 15:34:24.658878 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:24.658852    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:24.659037 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:24.658888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:24.659037 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:24.658941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:24.659037 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:24.658957    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:24.659037 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:24.659021    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:24.659282 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:24.659082    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:24.818618 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:24.818587    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="e79a56b842b548dcc25b62ad89c579613b5bc9fcea7a75b14e679b2f440af9d5" exitCode=0
Jul 06 15:34:24.818988 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:24.818682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"e79a56b842b548dcc25b62ad89c579613b5bc9fcea7a75b14e679b2f440af9d5"}
Jul 06 15:34:25.823353 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:25.823114    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="ad2ad92f6e0394d4b37c0c98cc5e4f3590b8411cdb9063272ac3eb681ae4d404" exitCode=0
Jul 06 15:34:25.823353 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:25.823192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"ad2ad92f6e0394d4b37c0c98cc5e4f3590b8411cdb9063272ac3eb681ae4d404"}
Jul 06 15:34:26.658178 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:26.658148    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:26.658333 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:26.658242    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:26.658408 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:26.658331    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:26.658408 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:26.658245    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:26.658408 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:26.658249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:26.658546 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:26.658429    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:27.243969 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:27.243937    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:27.244338 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.244084    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:27.244338 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.244134    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs podName:68ec5342-813b-40d3-8cd6-ed361ac72e0e nodeName:}" failed. No retries permitted until 2026-07-06 15:34:59.244119938 +0000 UTC m=+66.114167088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs") pod "network-metrics-daemon-btqvz" (UID: "68ec5342-813b-40d3-8cd6-ed361ac72e0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:34:27.345035 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:27.344995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:27.345217 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:27.345072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:27.345217 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345150    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:34:27.345217 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345179    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:34:27.345217 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345193    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dxgwl for pod openshift-network-diagnostics/network-check-target-vl8xx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:27.345217 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345202    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:27.345430 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345256    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret podName:eb24f93f-b357-4b8d-a604-6d89306a5dd6 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:59.345238248 +0000 UTC m=+66.215285411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret") pod "global-pull-secret-syncer-d48l4" (UID: "eb24f93f-b357-4b8d-a604-6d89306a5dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:34:27.345430 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:27.345277    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl podName:3d11e64e-27af-41a3-873c-997028e43d9f nodeName:}" failed. No retries permitted until 2026-07-06 15:34:59.345267346 +0000 UTC m=+66.215314494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dxgwl" (UniqueName: "kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl") pod "network-check-target-vl8xx" (UID: "3d11e64e-27af-41a3-873c-997028e43d9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:34:28.658847 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:28.658811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:28.658847 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:28.658844    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:28.659269 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:28.658891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:28.659269 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:28.658982    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-btqvz" podUID="68ec5342-813b-40d3-8cd6-ed361ac72e0e"
Jul 06 15:34:28.659269 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:28.659084    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl8xx" podUID="3d11e64e-27af-41a3-873c-997028e43d9f"
Jul 06 15:34:28.659269 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:28.659156    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d48l4" podUID="eb24f93f-b357-4b8d-a604-6d89306a5dd6"
Jul 06 15:34:29.395527 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.395451    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-21.ec2.internal" event="NodeReady"
Jul 06 15:34:29.395715 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.395625    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 15:34:29.454587 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.454547    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8gbx2"]
Jul 06 15:34:29.486345 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.486310    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-284nb"]
Jul 06 15:34:29.486499 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.486459    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.489595 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.489573    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 15:34:29.489727 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.489597    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 15:34:29.489727 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.489601    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8w9vb\""
Jul 06 15:34:29.508651 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.508622    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-284nb"]
Jul 06 15:34:29.508776 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.508678    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8gbx2"]
Jul 06 15:34:29.508776 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.508772    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.511616 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.511569    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pk7tw\""
Jul 06 15:34:29.511821 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.511796    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 15:34:29.511902 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.511828    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 15:34:29.511902 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.511854    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 15:34:29.561741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.561708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/1420a3f1-5a18-435f-bc21-6fe18f4f151a-kube-api-access-5xcmd\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.561891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.561778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1420a3f1-5a18-435f-bc21-6fe18f4f151a-config-volume\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.561891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.561810    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.561891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.561824    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1420a3f1-5a18-435f-bc21-6fe18f4f151a-tmp-dir\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.662718 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8pv\" (UniqueName: \"kubernetes.io/projected/0b0d376a-ab7d-4d85-a086-ec9220f2b072-kube-api-access-hv8pv\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.662718 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1420a3f1-5a18-435f-bc21-6fe18f4f151a-config-volume\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1420a3f1-5a18-435f-bc21-6fe18f4f151a-tmp-dir\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/1420a3f1-5a18-435f-bc21-6fe18f4f151a-kube-api-access-5xcmd\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.662922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b0d376a-ab7d-4d85-a086-ec9220f2b072-cert\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:29.662924    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:29.662976    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls podName:1420a3f1-5a18-435f-bc21-6fe18f4f151a nodeName:}" failed. No retries permitted until 2026-07-06 15:34:30.162961869 +0000 UTC m=+37.033009016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls") pod "dns-default-8gbx2" (UID: "1420a3f1-5a18-435f-bc21-6fe18f4f151a") : secret "dns-default-metrics-tls" not found
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.663140    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1420a3f1-5a18-435f-bc21-6fe18f4f151a-tmp-dir\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.663220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.663183    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1420a3f1-5a18-435f-bc21-6fe18f4f151a-config-volume\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.680162 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.680140    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/1420a3f1-5a18-435f-bc21-6fe18f4f151a-kube-api-access-5xcmd\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:29.764254 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.764216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b0d376a-ab7d-4d85-a086-ec9220f2b072-cert\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.764437 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.764268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8pv\" (UniqueName: \"kubernetes.io/projected/0b0d376a-ab7d-4d85-a086-ec9220f2b072-kube-api-access-hv8pv\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.767035 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.767007    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b0d376a-ab7d-4d85-a086-ec9220f2b072-cert\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.775711 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.775687    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8pv\" (UniqueName: \"kubernetes.io/projected/0b0d376a-ab7d-4d85-a086-ec9220f2b072-kube-api-access-hv8pv\") pod \"ingress-canary-284nb\" (UID: \"0b0d376a-ab7d-4d85-a086-ec9220f2b072\") " pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:29.819647 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:29.819611    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-284nb"
Jul 06 15:34:30.167758 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.167719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:30.167938 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:30.167868    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:34:30.167938 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:30.167932    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls podName:1420a3f1-5a18-435f-bc21-6fe18f4f151a nodeName:}" failed. No retries permitted until 2026-07-06 15:34:31.16791585 +0000 UTC m=+38.037963020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls") pod "dns-default-8gbx2" (UID: "1420a3f1-5a18-435f-bc21-6fe18f4f151a") : secret "dns-default-metrics-tls" not found
Jul 06 15:34:30.538950 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.538917    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-284nb"]
Jul 06 15:34:30.599569 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:30.599538    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0d376a_ab7d_4d85_a086_ec9220f2b072.slice/crio-5649404ae3ad535d65a0c982da8edaced2fa3083f75e36d0e383a96498445903 WatchSource:0}: Error finding container 5649404ae3ad535d65a0c982da8edaced2fa3083f75e36d0e383a96498445903: Status 404 returned error can't find the container with id 5649404ae3ad535d65a0c982da8edaced2fa3083f75e36d0e383a96498445903
Jul 06 15:34:30.658765 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.658739    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:30.658765 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.658757    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:30.658918 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.658744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:30.662921 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.662906    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 15:34:30.663449 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.662945    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w485s\""
Jul 06 15:34:30.663449 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.663329    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 15:34:30.663449 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.663382    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 15:34:30.663449 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.663399    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v99lt\""
Jul 06 15:34:30.663449 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.663435    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 15:34:30.836114 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.836080    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-284nb" event={"ID":"0b0d376a-ab7d-4d85-a086-ec9220f2b072","Type":"ContainerStarted","Data":"5649404ae3ad535d65a0c982da8edaced2fa3083f75e36d0e383a96498445903"}
Jul 06 15:34:30.839267 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:30.839226    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerStarted","Data":"c81510f9bcb7cd35e51496db2d63615345a584f2ec762852df770550b7b4b588"}
Jul 06 15:34:31.175468 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:31.175376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:31.175620 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:31.175562    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:34:31.175688 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:31.175644    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls podName:1420a3f1-5a18-435f-bc21-6fe18f4f151a nodeName:}" failed. No retries permitted until 2026-07-06 15:34:33.175621213 +0000 UTC m=+40.045668395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls") pod "dns-default-8gbx2" (UID: "1420a3f1-5a18-435f-bc21-6fe18f4f151a") : secret "dns-default-metrics-tls" not found
Jul 06 15:34:31.845617 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:31.845580    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="c81510f9bcb7cd35e51496db2d63615345a584f2ec762852df770550b7b4b588" exitCode=0
Jul 06 15:34:31.846148 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:31.845646    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"c81510f9bcb7cd35e51496db2d63615345a584f2ec762852df770550b7b4b588"}
Jul 06 15:34:32.289852 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.289812    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"]
Jul 06 15:34:32.306900 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.306874    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"
Jul 06 15:34:32.312049 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.312021    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sbbft\""
Jul 06 15:34:32.312517 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.312497    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 15:34:32.312623 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.312552    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 15:34:32.317374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.317346    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"]
Jul 06 15:34:32.383962 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.383920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fb4\" (UniqueName: \"kubernetes.io/projected/236e0826-227d-481b-82dc-c9416a8ee6f6-kube-api-access-m4fb4\") pod \"migrator-d57d58cc9-69845\" (UID: \"236e0826-227d-481b-82dc-c9416a8ee6f6\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"
Jul 06 15:34:32.485346 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.485303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fb4\" (UniqueName: \"kubernetes.io/projected/236e0826-227d-481b-82dc-c9416a8ee6f6-kube-api-access-m4fb4\") pod \"migrator-d57d58cc9-69845\" (UID: \"236e0826-227d-481b-82dc-c9416a8ee6f6\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"
Jul 06 15:34:32.498181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.498148    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fb4\" (UniqueName: \"kubernetes.io/projected/236e0826-227d-481b-82dc-c9416a8ee6f6-kube-api-access-m4fb4\") pod \"migrator-d57d58cc9-69845\" (UID: \"236e0826-227d-481b-82dc-c9416a8ee6f6\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"
Jul 06 15:34:32.617526 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.617439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"
Jul 06 15:34:32.851519 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.851483    2567 generic.go:356] "Generic (PLEG): container finished" podID="f73df3d9-c51a-4e9d-9dd0-1d5129116359" containerID="b3e2bcae2f9083f099f70bfc192c2527d511e7068c2e81d19527f28ad7a89da3" exitCode=0
Jul 06 15:34:32.851900 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.851549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerDied","Data":"b3e2bcae2f9083f099f70bfc192c2527d511e7068c2e81d19527f28ad7a89da3"}
Jul 06 15:34:32.942634 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.942446    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845"]
Jul 06 15:34:32.945071 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:32.945033    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236e0826_227d_481b_82dc_c9416a8ee6f6.slice/crio-d849ef20b70a74980d847fa24a745f9f9e5da0f85b47003261fd008aaee72355 WatchSource:0}: Error finding container d849ef20b70a74980d847fa24a745f9f9e5da0f85b47003261fd008aaee72355: Status 404 returned error can't find the container with id d849ef20b70a74980d847fa24a745f9f9e5da0f85b47003261fd008aaee72355
Jul 06 15:34:32.985733 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:32.985710    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k2d54"]
Jul 06 15:34:33.012564 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.012541    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2d54"]
Jul 06 15:34:33.012696 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.012657    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.019011 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.018989    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 15:34:33.019100 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.019080    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rtrq\""
Jul 06 15:34:33.019345 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.019327    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 15:34:33.019345 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.019338    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 15:34:33.019656 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.019642    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 15:34:33.090132 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.090104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25e27276-9bc9-4f81-85ae-5c2fce17e039-crio-socket\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.090132 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.090136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.090324 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.090158    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e27276-9bc9-4f81-85ae-5c2fce17e039-data-volume\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.090324 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.090242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.090324 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.090275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6kw\" (UniqueName: \"kubernetes.io/projected/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-api-access-nn6kw\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191542 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:33.191630 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191630 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191569    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6kw\" (UniqueName: \"kubernetes.io/projected/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-api-access-nn6kw\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191748 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25e27276-9bc9-4f81-85ae-5c2fce17e039-crio-socket\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191748 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191685    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191748 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191701    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:34:33.191748 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191703    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:34:33.191748 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e27276-9bc9-4f81-85ae-5c2fce17e039-data-volume\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.191958 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191770    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls podName:1420a3f1-5a18-435f-bc21-6fe18f4f151a nodeName:}" failed. No retries permitted until 2026-07-06 15:34:37.191747852 +0000 UTC m=+44.061795012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls") pod "dns-default-8gbx2" (UID: "1420a3f1-5a18-435f-bc21-6fe18f4f151a") : secret "dns-default-metrics-tls" not found
Jul 06 15:34:33.191958 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191790    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:33.691780537 +0000 UTC m=+40.561827687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : configmap "kube-rbac-proxy" not found
Jul 06 15:34:33.191958 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191813    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:33.191958 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.191866    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:33.691849865 +0000 UTC m=+40.561897017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:33.192135 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.191964    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25e27276-9bc9-4f81-85ae-5c2fce17e039-crio-socket\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.192135 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.192028    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e27276-9bc9-4f81-85ae-5c2fce17e039-data-volume\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.202516 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.202491    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6kw\" (UniqueName: \"kubernetes.io/projected/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-api-access-nn6kw\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.695312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.695276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.695458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.695350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:33.695458 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.695403    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:34:33.695518 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.695454    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:33.695518 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.695466    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:34.695451416 +0000 UTC m=+41.565498562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : configmap "kube-rbac-proxy" not found
Jul 06 15:34:33.695518 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:33.695507    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:34.695493116 +0000 UTC m=+41.565540265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:33.857699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.857636    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" event={"ID":"f73df3d9-c51a-4e9d-9dd0-1d5129116359","Type":"ContainerStarted","Data":"36c64b4971301a57f2e039de614ab7a3211b6f2da9cb3c3664ec5b83e0c51633"}
Jul 06 15:34:33.859143 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.859117    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-284nb" event={"ID":"0b0d376a-ab7d-4d85-a086-ec9220f2b072","Type":"ContainerStarted","Data":"7d9bad52da5dabc4d312321d22828cde7e9846354b0c2faeff7893c10eeeff09"}
Jul 06 15:34:33.860131 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.860106    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845" event={"ID":"236e0826-227d-481b-82dc-c9416a8ee6f6","Type":"ContainerStarted","Data":"d849ef20b70a74980d847fa24a745f9f9e5da0f85b47003261fd008aaee72355"}
Jul 06 15:34:33.885884 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.885835    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t2b2d" podStartSLOduration=9.500882204 podStartE2EDuration="40.885820533s" totalImagesPullingTime="31.384938329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:33.88465376 +0000 UTC m=+40.754700928" watchObservedRunningTime="2026-07-06 15:34:33.885820533 +0000 UTC m=+40.755867701"
Jul 06 15:34:33.906711 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:33.906635    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-284nb" podStartSLOduration=2.352427374 podStartE2EDuration="4.906619975s" totalImagesPullingTime="2.554192601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:33.905746972 +0000 UTC m=+40.775794140" watchObservedRunningTime="2026-07-06 15:34:33.906619975 +0000 UTC m=+40.776667144"
Jul 06 15:34:34.705088 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:34.705048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:34.705252 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:34.705111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:34.705252 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:34.705178    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:34:34.705252 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:34.705209    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:34.705252 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:34.705241    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.705225601 +0000 UTC m=+43.575272754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : configmap "kube-rbac-proxy" not found
Jul 06 15:34:34.705386 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:34.705263    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.705249298 +0000 UTC m=+43.575296443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:35.665974 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:35.665952    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5ts55_2cb8845b-4fb5-40c1-b7ae-399f78bf1d94/dns-node-resolver/0.log"
Jul 06 15:34:35.866520 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:35.866486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845" event={"ID":"236e0826-227d-481b-82dc-c9416a8ee6f6","Type":"ContainerStarted","Data":"fb78445b99a18dd9d964d3926bd85cf0eeee627a26bad2108d8ce5828af4a62c"}
Jul 06 15:34:35.866655 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:35.866538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845" event={"ID":"236e0826-227d-481b-82dc-c9416a8ee6f6","Type":"ContainerStarted","Data":"9d76f4b9248b4a2794e385bac6b10707330de77bb931f4636c950c2be7d272bc"}
Jul 06 15:34:36.722245 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:36.722210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:36.722602 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:36.722265    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:36.722602 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:36.722335    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:34:36.722602 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:36.722364    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:36.722602 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:36.722384    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:40.722371531 +0000 UTC m=+47.592418680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : configmap "kube-rbac-proxy" not found
Jul 06 15:34:36.722602 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:36.722415    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:40.722402114 +0000 UTC m=+47.592449282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:37.072528 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:37.072453    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7gdp_c60ca798-3c9a-48dd-8565-fd717dadb1e1/node-ca/0.log"
Jul 06 15:34:37.226297 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:37.226257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:37.226483 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:37.226395    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:34:37.226483 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:37.226456    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls podName:1420a3f1-5a18-435f-bc21-6fe18f4f151a nodeName:}" failed. No retries permitted until 2026-07-06 15:34:45.226441789 +0000 UTC m=+52.096488935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls") pod "dns-default-8gbx2" (UID: "1420a3f1-5a18-435f-bc21-6fe18f4f151a") : secret "dns-default-metrics-tls" not found
Jul 06 15:34:37.469732 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:37.469698    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-284nb_0b0d376a-ab7d-4d85-a086-ec9220f2b072/serve-healthcheck-canary/0.log"
Jul 06 15:34:38.071272 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:38.071234    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-69845_236e0826-227d-481b-82dc-c9416a8ee6f6/migrator/0.log"
Jul 06 15:34:38.265696 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:38.265651    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-69845_236e0826-227d-481b-82dc-c9416a8ee6f6/graceful-termination/0.log"
Jul 06 15:34:40.752805 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:40.752760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:40.753275 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:40.752822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:40.753275 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:40.752897    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:34:40.753275 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:40.752913    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:40.753275 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:40.752951    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:48.752937458 +0000 UTC m=+55.622984604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : configmap "kube-rbac-proxy" not found
Jul 06 15:34:40.753275 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:40.752968    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:48.752956461 +0000 UTC m=+55.623003631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:45.288752 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.288706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:45.291320 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.291298    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1420a3f1-5a18-435f-bc21-6fe18f4f151a-metrics-tls\") pod \"dns-default-8gbx2\" (UID: \"1420a3f1-5a18-435f-bc21-6fe18f4f151a\") " pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:45.397731 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.397688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:45.543898 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.543772    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-69845" podStartSLOduration=11.376583991 podStartE2EDuration="13.543737275s" totalImagesPullingTime="2.167153284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:35.900980463 +0000 UTC m=+42.771027631" watchObservedRunningTime="2026-07-06 15:34:45.543737275 +0000 UTC m=+52.413784443"
Jul 06 15:34:45.545106 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.545084    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8gbx2"]
Jul 06 15:34:45.548876 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:45.548851    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1420a3f1_5a18_435f_bc21_6fe18f4f151a.slice/crio-806a2889a796fe05b96df52e8b91f885b3fba2d41a3b788c0e68fae1c9efa603 WatchSource:0}: Error finding container 806a2889a796fe05b96df52e8b91f885b3fba2d41a3b788c0e68fae1c9efa603: Status 404 returned error can't find the container with id 806a2889a796fe05b96df52e8b91f885b3fba2d41a3b788c0e68fae1c9efa603
Jul 06 15:34:45.892499 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:45.892410    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8gbx2" event={"ID":"1420a3f1-5a18-435f-bc21-6fe18f4f151a","Type":"ContainerStarted","Data":"806a2889a796fe05b96df52e8b91f885b3fba2d41a3b788c0e68fae1c9efa603"}
Jul 06 15:34:48.814046 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.814008    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:48.814417 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.814061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:48.814417 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:48.814161    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:48.814417 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:34:48.814226    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls podName:25e27276-9bc9-4f81-85ae-5c2fce17e039 nodeName:}" failed. No retries permitted until 2026-07-06 15:35:04.81420913 +0000 UTC m=+71.684256276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2d54" (UID: "25e27276-9bc9-4f81-85ae-5c2fce17e039") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:34:48.814559 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.814541    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25e27276-9bc9-4f81-85ae-5c2fce17e039-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:34:48.901465 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.901431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8gbx2" event={"ID":"1420a3f1-5a18-435f-bc21-6fe18f4f151a","Type":"ContainerStarted","Data":"504006fc0a7e2b0c1e60f474b1e0847ee86d11f44268c250bed3a522c8374324"}
Jul 06 15:34:48.901650 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.901490    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8gbx2" event={"ID":"1420a3f1-5a18-435f-bc21-6fe18f4f151a","Type":"ContainerStarted","Data":"587eb9d4f1a85c45ecff1be9b8b6c51ea2b015df618ac1089a0500cd95f9e0fb"}
Jul 06 15:34:48.901650 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:48.901555    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:54.836627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:54.836598    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-488fh"
Jul 06 15:34:54.901523 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:54.901455    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8gbx2" podStartSLOduration=23.457094284 podStartE2EDuration="25.901440379s" totalImagesPullingTime="2.444346095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:48.930301281 +0000 UTC m=+55.800348450" watchObservedRunningTime="2026-07-06 15:34:54.901440379 +0000 UTC m=+61.771487544"
Jul 06 15:34:55.642153 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.642117    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"]
Jul 06 15:34:55.644527 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.644510    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.650182 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.650155    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 15:34:55.650906 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.650886    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 15:34:55.651009 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.650912    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 15:34:55.651009 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.650984    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 15:34:55.651214 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.651201    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 15:34:55.651266 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.651244    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 15:34:55.660182 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5j5\" (UniqueName: \"kubernetes.io/projected/96489f0f-e8cd-497a-82a0-278f38ffd363-kube-api-access-qf5j5\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.660182 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.660182 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.660182 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660150    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96489f0f-e8cd-497a-82a0-278f38ffd363-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.660428 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.660428 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.662968 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.660734    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 15:34:55.692416 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.692392    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"]
Jul 06 15:34:55.761183 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5j5\" (UniqueName: \"kubernetes.io/projected/96489f0f-e8cd-497a-82a0-278f38ffd363-kube-api-access-qf5j5\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.761322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.761322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.761322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96489f0f-e8cd-497a-82a0-278f38ffd363-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.761322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.761483 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.761340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.762070 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.762038    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96489f0f-e8cd-497a-82a0-278f38ffd363-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.763833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.763799    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.763957 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.763938    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-ca\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.764363 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.764344    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.764604 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.764584    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96489f0f-e8cd-497a-82a0-278f38ffd363-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.784251 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.784226    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5j5\" (UniqueName: \"kubernetes.io/projected/96489f0f-e8cd-497a-82a0-278f38ffd363-kube-api-access-qf5j5\") pod \"cluster-proxy-proxy-agent-8569578555-gdbwq\" (UID: \"96489f0f-e8cd-497a-82a0-278f38ffd363\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.835572 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.835548    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-df74c4f86-5mjbl"]
Jul 06 15:34:55.837703 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.837688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.840815 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.840798    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 15:34:55.841590 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.841571    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-98ctb\""
Jul 06 15:34:55.841869 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.841854    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 15:34:55.842358 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.842340    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 15:34:55.849982 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.849965    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 15:34:55.862201 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a995b57-0715-428d-ad2e-05a005ec8aae-ca-trust-extracted\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862279 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-image-registry-private-configuration\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862279 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-tls\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862279 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-trusted-ca\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862372 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862293    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-bound-sa-token\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862372 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-installation-pull-secrets\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862433 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-certificates\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.862433 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.862403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92mqk\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-kube-api-access-92mqk\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.888202 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.888176    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df74c4f86-5mjbl"]
Jul 06 15:34:55.963601 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-image-registry-private-configuration\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963601 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-tls\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963842 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-trusted-ca\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963842 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-bound-sa-token\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963842 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-installation-pull-secrets\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963988 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-certificates\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963988 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963887    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92mqk\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-kube-api-access-92mqk\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.963988 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.963971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a995b57-0715-428d-ad2e-05a005ec8aae-ca-trust-extracted\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.964414 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.964314    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a995b57-0715-428d-ad2e-05a005ec8aae-ca-trust-extracted\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.964703 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.964685    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-certificates\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.964982 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.964959    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a995b57-0715-428d-ad2e-05a005ec8aae-trusted-ca\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.965975 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.965951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"
Jul 06 15:34:55.966179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.966160    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-image-registry-private-configuration\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.966297 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.966280    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-registry-tls\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.966372 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.966352    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a995b57-0715-428d-ad2e-05a005ec8aae-installation-pull-secrets\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.987249 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.987225    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92mqk\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-kube-api-access-92mqk\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:55.987387 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:55.987276    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a995b57-0715-428d-ad2e-05a005ec8aae-bound-sa-token\") pod \"image-registry-df74c4f86-5mjbl\" (UID: \"7a995b57-0715-428d-ad2e-05a005ec8aae\") " pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:56.117456 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.117426    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq"]
Jul 06 15:34:56.120501 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:56.120473    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96489f0f_e8cd_497a_82a0_278f38ffd363.slice/crio-a1aceb35b15846208f32e173bce54e3ec4035c3d7b0b98e4316ee48810b86dcd WatchSource:0}: Error finding container a1aceb35b15846208f32e173bce54e3ec4035c3d7b0b98e4316ee48810b86dcd: Status 404 returned error can't find the container with id a1aceb35b15846208f32e173bce54e3ec4035c3d7b0b98e4316ee48810b86dcd
Jul 06 15:34:56.146340 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.146321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:56.281389 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.281359    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df74c4f86-5mjbl"]
Jul 06 15:34:56.284122 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:56.284095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a995b57_0715_428d_ad2e_05a005ec8aae.slice/crio-4cd8b306a3d17be2f997e9a148da7b095c4a5dcd62b7b5fff57d9ed9f153de5d WatchSource:0}: Error finding container 4cd8b306a3d17be2f997e9a148da7b095c4a5dcd62b7b5fff57d9ed9f153de5d: Status 404 returned error can't find the container with id 4cd8b306a3d17be2f997e9a148da7b095c4a5dcd62b7b5fff57d9ed9f153de5d
Jul 06 15:34:56.922560 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.922520    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq" event={"ID":"96489f0f-e8cd-497a-82a0-278f38ffd363","Type":"ContainerStarted","Data":"a1aceb35b15846208f32e173bce54e3ec4035c3d7b0b98e4316ee48810b86dcd"}
Jul 06 15:34:56.924503 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.924224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl" event={"ID":"7a995b57-0715-428d-ad2e-05a005ec8aae","Type":"ContainerStarted","Data":"e4fba526494dd9d07ab862483f5d571d75b44b8450a8e849e803f129898021ba"}
Jul 06 15:34:56.924503 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.924275    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl" event={"ID":"7a995b57-0715-428d-ad2e-05a005ec8aae","Type":"ContainerStarted","Data":"4cd8b306a3d17be2f997e9a148da7b095c4a5dcd62b7b5fff57d9ed9f153de5d"}
Jul 06 15:34:56.924503 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.924310    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:34:56.953235 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:56.953191    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl" podStartSLOduration=1.9531767960000002 podStartE2EDuration="1.953176796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:56.95159955 +0000 UTC m=+63.821646718" watchObservedRunningTime="2026-07-06 15:34:56.953176796 +0000 UTC m=+63.823223963"
Jul 06 15:34:58.906013 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:58.905981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8gbx2"
Jul 06 15:34:59.295982 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.295947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:59.301282 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.301259    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 15:34:59.309337 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.309305    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec5342-813b-40d3-8cd6-ed361ac72e0e-metrics-certs\") pod \"network-metrics-daemon-btqvz\" (UID: \"68ec5342-813b-40d3-8cd6-ed361ac72e0e\") " pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:59.396785 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.396742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:59.396976 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.396828    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:59.400854 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.400828    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 15:34:59.400961 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.400828    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 15:34:59.410241 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.410214    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb24f93f-b357-4b8d-a604-6d89306a5dd6-original-pull-secret\") pod \"global-pull-secret-syncer-d48l4\" (UID: \"eb24f93f-b357-4b8d-a604-6d89306a5dd6\") " pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:59.410358 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.410228    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 15:34:59.421367 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.421266    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgwl\" (UniqueName: \"kubernetes.io/projected/3d11e64e-27af-41a3-873c-997028e43d9f-kube-api-access-dxgwl\") pod \"network-check-target-vl8xx\" (UID: \"3d11e64e-27af-41a3-873c-997028e43d9f\") " pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:59.473354 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.473313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v99lt\""
Jul 06 15:34:59.479723 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.479699    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w485s\""
Jul 06 15:34:59.479882 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.479803    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-btqvz"
Jul 06 15:34:59.487315 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.487280    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:34:59.487479 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.487462    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d48l4"
Jul 06 15:34:59.652244 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.652212    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d48l4"]
Jul 06 15:34:59.655753 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:59.655728    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb24f93f_b357_4b8d_a604_6d89306a5dd6.slice/crio-9bd7d5b8f8df45c98039397897e90f07a1464608a7b3656cd0b01f74d988b0e0 WatchSource:0}: Error finding container 9bd7d5b8f8df45c98039397897e90f07a1464608a7b3656cd0b01f74d988b0e0: Status 404 returned error can't find the container with id 9bd7d5b8f8df45c98039397897e90f07a1464608a7b3656cd0b01f74d988b0e0
Jul 06 15:34:59.670827 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.670778    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-btqvz"]
Jul 06 15:34:59.672788 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.672749    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vl8xx"]
Jul 06 15:34:59.674605 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:59.674541    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ec5342_813b_40d3_8cd6_ed361ac72e0e.slice/crio-cf65000f0c1699236fdf6e52a72afa842abd6d462fa59d52397aea06dce0f9ea WatchSource:0}: Error finding container cf65000f0c1699236fdf6e52a72afa842abd6d462fa59d52397aea06dce0f9ea: Status 404 returned error can't find the container with id cf65000f0c1699236fdf6e52a72afa842abd6d462fa59d52397aea06dce0f9ea
Jul 06 15:34:59.678101 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:34:59.678064    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d11e64e_27af_41a3_873c_997028e43d9f.slice/crio-aa7ad06a11cd14c5ebdca9aee39cf62e55d98feb16d09ad5d6301c5408964a5b WatchSource:0}: Error finding container aa7ad06a11cd14c5ebdca9aee39cf62e55d98feb16d09ad5d6301c5408964a5b: Status 404 returned error can't find the container with id aa7ad06a11cd14c5ebdca9aee39cf62e55d98feb16d09ad5d6301c5408964a5b
Jul 06 15:34:59.934992 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.934911    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-btqvz" event={"ID":"68ec5342-813b-40d3-8cd6-ed361ac72e0e","Type":"ContainerStarted","Data":"cf65000f0c1699236fdf6e52a72afa842abd6d462fa59d52397aea06dce0f9ea"}
Jul 06 15:34:59.937017 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.936993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq" event={"ID":"96489f0f-e8cd-497a-82a0-278f38ffd363","Type":"ContainerStarted","Data":"69f0ed2a8962ab1cbe13d544289ed114d3733a68e8d815fa89bbd88a1aa71da8"}
Jul 06 15:34:59.937155 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.937028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq" event={"ID":"96489f0f-e8cd-497a-82a0-278f38ffd363","Type":"ContainerStarted","Data":"26a2b464d2d005cdaa294fddd420fe038e9853450d939f6414175ba73fa011f6"}
Jul 06 15:34:59.937155 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.937047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq" event={"ID":"96489f0f-e8cd-497a-82a0-278f38ffd363","Type":"ContainerStarted","Data":"42351eea314a292f466abb307c6f37ff519a0734977b0565e6c58a7129905b54"}
Jul 06 15:34:59.938254 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.938234    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vl8xx" event={"ID":"3d11e64e-27af-41a3-873c-997028e43d9f","Type":"ContainerStarted","Data":"aa7ad06a11cd14c5ebdca9aee39cf62e55d98feb16d09ad5d6301c5408964a5b"}
Jul 06 15:34:59.939305 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.939284    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d48l4" event={"ID":"eb24f93f-b357-4b8d-a604-6d89306a5dd6","Type":"ContainerStarted","Data":"9bd7d5b8f8df45c98039397897e90f07a1464608a7b3656cd0b01f74d988b0e0"}
Jul 06 15:34:59.971627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:34:59.969787    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8569578555-gdbwq" podStartSLOduration=1.5326271280000001 podStartE2EDuration="4.969766632s" totalImagesPullingTime="3.437139504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:59.968519668 +0000 UTC m=+66.838566837" watchObservedRunningTime="2026-07-06 15:34:59.969766632 +0000 UTC m=+66.839813801"
Jul 06 15:35:01.951111 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:01.951069    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-btqvz" event={"ID":"68ec5342-813b-40d3-8cd6-ed361ac72e0e","Type":"ContainerStarted","Data":"0354dca1cea48dd622461a5e7c6a252c8ff0b1d8aea67f90257fb0c92b19e0ca"}
Jul 06 15:35:01.951627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:01.951141    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-btqvz" event={"ID":"68ec5342-813b-40d3-8cd6-ed361ac72e0e","Type":"ContainerStarted","Data":"8a42ccdd837b54901d9ae0b62294ea16f685b06aed5c152d5d6d9dab1d4b5c04"}
Jul 06 15:35:01.977277 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:01.977215    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-btqvz" podStartSLOduration=67.523548557 podStartE2EDuration="1m8.977194012s" totalImagesPullingTime="1.453645455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:01.975656122 +0000 UTC m=+68.845703291" watchObservedRunningTime="2026-07-06 15:35:01.977194012 +0000 UTC m=+68.847241181"
Jul 06 15:35:04.838949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.838910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:35:04.841318 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.841288    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25e27276-9bc9-4f81-85ae-5c2fce17e039-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2d54\" (UID: \"25e27276-9bc9-4f81-85ae-5c2fce17e039\") " pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:35:04.963179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.963144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vl8xx" event={"ID":"3d11e64e-27af-41a3-873c-997028e43d9f","Type":"ContainerStarted","Data":"dc5983678595087e1aaefcb1251fa75cf9ed8edd971eff111c55080031cfc980"}
Jul 06 15:35:04.963385 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.963236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:35:04.964612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.964582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d48l4" event={"ID":"eb24f93f-b357-4b8d-a604-6d89306a5dd6","Type":"ContainerStarted","Data":"5026af0285a17d966f4e2cfc5cf4055d8c5801e747dba8bd9a8ba310fa2f2da7"}
Jul 06 15:35:04.982516 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.982471    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vl8xx" podStartSLOduration=67.484135327 podStartE2EDuration="1m11.98245626s" totalImagesPullingTime="4.498320933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:04.981418135 +0000 UTC m=+71.851465304" watchObservedRunningTime="2026-07-06 15:35:04.98245626 +0000 UTC m=+71.852503427"
Jul 06 15:35:04.998515 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:04.998456    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d48l4" podStartSLOduration=67.475360846 podStartE2EDuration="1m11.998440851s" totalImagesPullingTime="4.523080005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:04.997750675 +0000 UTC m=+71.867797842" watchObservedRunningTime="2026-07-06 15:35:04.998440851 +0000 UTC m=+71.868488063"
Jul 06 15:35:05.124252 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:05.124179    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rtrq\""
Jul 06 15:35:05.132062 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:05.132038    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2d54"
Jul 06 15:35:05.261418 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:05.261375    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2d54"]
Jul 06 15:35:05.264363 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:35:05.264335    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e27276_9bc9_4f81_85ae_5c2fce17e039.slice/crio-597ff9d7a4d21a5a92a1290fbd5c1bcfc4b061463b46b2df878fdfe0a08ad59e WatchSource:0}: Error finding container 597ff9d7a4d21a5a92a1290fbd5c1bcfc4b061463b46b2df878fdfe0a08ad59e: Status 404 returned error can't find the container with id 597ff9d7a4d21a5a92a1290fbd5c1bcfc4b061463b46b2df878fdfe0a08ad59e
Jul 06 15:35:05.972249 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:05.972175    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2d54" event={"ID":"25e27276-9bc9-4f81-85ae-5c2fce17e039","Type":"ContainerStarted","Data":"597ff9d7a4d21a5a92a1290fbd5c1bcfc4b061463b46b2df878fdfe0a08ad59e"}
Jul 06 15:35:08.983411 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:08.983374    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2d54" event={"ID":"25e27276-9bc9-4f81-85ae-5c2fce17e039","Type":"ContainerStarted","Data":"c15e7425b5d38be7c21859180107543158f761e7ffe9fac493407d72cf1a8927"}
Jul 06 15:35:09.989338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:09.989300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2d54" event={"ID":"25e27276-9bc9-4f81-85ae-5c2fce17e039","Type":"ContainerStarted","Data":"99688ab2411e00fb8b672a07d8b2b2a2fb9e348d8272070b8c0d0c0512102775"}
Jul 06 15:35:13.001078 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:13.001031    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2d54" event={"ID":"25e27276-9bc9-4f81-85ae-5c2fce17e039","Type":"ContainerStarted","Data":"b70e04ad907c45b31c2360fa329a96e5ef3644d2fdd1f35a05f673c37ec4b743"}
Jul 06 15:35:13.033376 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:13.033322    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k2d54" podStartSLOduration=34.453444798 podStartE2EDuration="41.033307299s" totalImagesPullingTime="6.579862501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:34:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:13.032338217 +0000 UTC m=+79.902385384" watchObservedRunningTime="2026-07-06 15:35:13.033307299 +0000 UTC m=+79.903354466"
Jul 06 15:35:14.289064 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.288951    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cvf4k"]
Jul 06 15:35:14.295578 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.295554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.299935 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.299913    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 15:35:14.299935 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.299941    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 15:35:14.300168 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.299975    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 15:35:14.300168 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.299926    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 15:35:14.300168 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.299924    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 15:35:14.300782 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.300758    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7t9r2\""
Jul 06 15:35:14.301011 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.300990    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 15:35:14.302234 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.302214    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"]
Jul 06 15:35:14.304465 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.304447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.306886 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.306870    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 06 15:35:14.307284 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.307264    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cxmj2\""
Jul 06 15:35:14.307481 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.307464    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 06 15:35:14.307552 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.307471    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 06 15:35:14.329492 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.329461    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"]
Jul 06 15:35:14.415993 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.415967    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-sys\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416174 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416174 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-root\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416174 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87e1bded-e130-47fd-acde-9860ce5d753d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-tls\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-wtmp\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416339 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmsq\" (UniqueName: \"kubernetes.io/projected/87e1bded-e130-47fd-acde-9860ce5d753d-kube-api-access-kdmsq\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416353    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-metrics-client-ca\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-textfile\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.416536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.416487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2ksr\" (UniqueName: \"kubernetes.io/projected/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-kube-api-access-d2ksr\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517208 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-sys\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517208 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-root\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87e1bded-e130-47fd-acde-9860ce5d753d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517301    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-sys\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517333    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-tls\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517311    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-root\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-wtmp\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmsq\" (UniqueName: \"kubernetes.io/projected/87e1bded-e130-47fd-acde-9860ce5d753d-kube-api-access-kdmsq\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517565    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-wtmp\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517580    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87e1bded-e130-47fd-acde-9860ce5d753d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.517781 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.518109 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.518109 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-metrics-client-ca\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.518109 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-textfile\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.518109 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.517904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ksr\" (UniqueName: \"kubernetes.io/projected/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-kube-api-access-d2ksr\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.518307 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.518294    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.518471 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.518443    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.518546 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.518472    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-metrics-client-ca\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.518546 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.518498    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-textfile\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.518746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.518728    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.520067 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.520038    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.520542 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.520522    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.520610 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.520540    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e1bded-e130-47fd-acde-9860ce5d753d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.520610 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.520552    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-node-exporter-tls\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.528788 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.528762    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ksr\" (UniqueName: \"kubernetes.io/projected/9c0430c8-7e38-49ba-84b4-955d2b31b9d8-kube-api-access-d2ksr\") pod \"node-exporter-cvf4k\" (UID: \"9c0430c8-7e38-49ba-84b4-955d2b31b9d8\") " pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.529222 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.529200    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmsq\" (UniqueName: \"kubernetes.io/projected/87e1bded-e130-47fd-acde-9860ce5d753d-kube-api-access-kdmsq\") pod \"kube-state-metrics-7c9dbb8b7d-dgn8m\" (UID: \"87e1bded-e130-47fd-acde-9860ce5d753d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.604730 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.604641    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cvf4k"
Jul 06 15:35:14.612736 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.612692    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"
Jul 06 15:35:14.615465 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:35:14.615443    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0430c8_7e38_49ba_84b4_955d2b31b9d8.slice/crio-bc8d9defd21912a01b526da72c32531d25c0c674215ec077594e104fd0cca6cd WatchSource:0}: Error finding container bc8d9defd21912a01b526da72c32531d25c0c674215ec077594e104fd0cca6cd: Status 404 returned error can't find the container with id bc8d9defd21912a01b526da72c32531d25c0c674215ec077594e104fd0cca6cd
Jul 06 15:35:14.746867 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:14.746834    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m"]
Jul 06 15:35:14.750159 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:35:14.750127    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e1bded_e130_47fd_acde_9860ce5d753d.slice/crio-35149f7570d2d659c3f1c658c31707023b790ad3db55f4bb42500af601331405 WatchSource:0}: Error finding container 35149f7570d2d659c3f1c658c31707023b790ad3db55f4bb42500af601331405: Status 404 returned error can't find the container with id 35149f7570d2d659c3f1c658c31707023b790ad3db55f4bb42500af601331405
Jul 06 15:35:15.009159 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:15.009120    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvf4k" event={"ID":"9c0430c8-7e38-49ba-84b4-955d2b31b9d8","Type":"ContainerStarted","Data":"bc8d9defd21912a01b526da72c32531d25c0c674215ec077594e104fd0cca6cd"}
Jul 06 15:35:15.010249 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:15.010213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m" event={"ID":"87e1bded-e130-47fd-acde-9860ce5d753d","Type":"ContainerStarted","Data":"35149f7570d2d659c3f1c658c31707023b790ad3db55f4bb42500af601331405"}
Jul 06 15:35:16.016713 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:16.016604    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c0430c8-7e38-49ba-84b4-955d2b31b9d8" containerID="51601b69488a1194000d075ac8d43cbded68fd0ceace73c1d482a08f9d40c764" exitCode=0
Jul 06 15:35:16.016713 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:16.016658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvf4k" event={"ID":"9c0430c8-7e38-49ba-84b4-955d2b31b9d8","Type":"ContainerDied","Data":"51601b69488a1194000d075ac8d43cbded68fd0ceace73c1d482a08f9d40c764"}
Jul 06 15:35:17.022360 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.022321    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvf4k" event={"ID":"9c0430c8-7e38-49ba-84b4-955d2b31b9d8","Type":"ContainerStarted","Data":"1f9b036d727bc4e5d52823981e76832409f436443791d384d06be7b964118750"}
Jul 06 15:35:17.022822 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.022425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cvf4k" event={"ID":"9c0430c8-7e38-49ba-84b4-955d2b31b9d8","Type":"ContainerStarted","Data":"6b1c26aed95f79d26bb632007688bcc5d770842b3abfc9bdfaac438102038263"}
Jul 06 15:35:17.024331 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.024302    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m" event={"ID":"87e1bded-e130-47fd-acde-9860ce5d753d","Type":"ContainerStarted","Data":"85fc339e435b0b7264a44333d6da15a06e78db7851b5c25042d41fbc1c2eeb14"}
Jul 06 15:35:17.024445 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.024358    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m" event={"ID":"87e1bded-e130-47fd-acde-9860ce5d753d","Type":"ContainerStarted","Data":"5658d19637187a06b53909648ac38990b88519802fac8c530868494cf26b4df9"}
Jul 06 15:35:17.024445 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.024379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m" event={"ID":"87e1bded-e130-47fd-acde-9860ce5d753d","Type":"ContainerStarted","Data":"d4dbb9f54068a8fa0818e662ca61758f362e8cf3db4de97a720d4a997a1dc050"}
Jul 06 15:35:17.162533 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.162475    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cvf4k" podStartSLOduration=1.983298987 podStartE2EDuration="3.162458167s" totalImagesPullingTime="1.10341418s" totalInitContainerRuntime="75.745ms" isStatefulPod=true podCreationTimestamp="2026-07-06 15:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:17.097988414 +0000 UTC m=+83.968035581" watchObservedRunningTime="2026-07-06 15:35:17.162458167 +0000 UTC m=+84.032505334"
Jul 06 15:35:17.162760 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.162583    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dgn8m" podStartSLOduration=1.694028035 podStartE2EDuration="3.162578054s" totalImagesPullingTime="1.468550019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:17.161298713 +0000 UTC m=+84.031345892" watchObservedRunningTime="2026-07-06 15:35:17.162578054 +0000 UTC m=+84.032625222"
Jul 06 15:35:17.932535 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:17.932503    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-df74c4f86-5mjbl"
Jul 06 15:35:19.034015 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.033981    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"]
Jul 06 15:35:19.036748 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.036733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.045128 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.045109    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d8ppf\""
Jul 06 15:35:19.045280 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.045263    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 15:35:19.064771 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.064745    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"]
Jul 06 15:35:19.156493 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.156457    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bsr\" (UniqueName: \"kubernetes.io/projected/16183bad-7781-4738-a10e-989bcc171f4e-kube-api-access-56bsr\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.156657 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.156511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.257032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.256995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bsr\" (UniqueName: \"kubernetes.io/projected/16183bad-7781-4738-a10e-989bcc171f4e-kube-api-access-56bsr\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.257198 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.257048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.257198 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:35:19.257163    2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 06 15:35:19.257266 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:35:19.257223    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert podName:16183bad-7781-4738-a10e-989bcc171f4e nodeName:}" failed. No retries permitted until 2026-07-06 15:35:19.75720592 +0000 UTC m=+86.627253067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-mb8wc" (UID: "16183bad-7781-4738-a10e-989bcc171f4e") : secret "monitoring-plugin-cert" not found
Jul 06 15:35:19.270511 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.270476    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bsr\" (UniqueName: \"kubernetes.io/projected/16183bad-7781-4738-a10e-989bcc171f4e-kube-api-access-56bsr\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.759464 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.759430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.762056 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.762023    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16183bad-7781-4738-a10e-989bcc171f4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-mb8wc\" (UID: \"16183bad-7781-4738-a10e-989bcc171f4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:19.945306 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:19.945261    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:20.098382 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:20.098349    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"]
Jul 06 15:35:20.101518 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:35:20.101493    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16183bad_7781_4738_a10e_989bcc171f4e.slice/crio-d3334ff2cdfe3ee44456c4291e04b9279918e594c8b9e2be3afe60b8e2968eb0 WatchSource:0}: Error finding container d3334ff2cdfe3ee44456c4291e04b9279918e594c8b9e2be3afe60b8e2968eb0: Status 404 returned error can't find the container with id d3334ff2cdfe3ee44456c4291e04b9279918e594c8b9e2be3afe60b8e2968eb0
Jul 06 15:35:21.040547 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:21.040506    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc" event={"ID":"16183bad-7781-4738-a10e-989bcc171f4e","Type":"ContainerStarted","Data":"d3334ff2cdfe3ee44456c4291e04b9279918e594c8b9e2be3afe60b8e2968eb0"}
Jul 06 15:35:22.045533 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:22.045438    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc" event={"ID":"16183bad-7781-4738-a10e-989bcc171f4e","Type":"ContainerStarted","Data":"08144eb2a884ed1f46a67b20f8c5328f9c26f827330b7445bc146d05ba2a03c5"}
Jul 06 15:35:22.045968 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:22.045622    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:22.050446 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:22.050424    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc"
Jul 06 15:35:22.069298 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:22.069248    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-mb8wc" podStartSLOduration=1.443620565 podStartE2EDuration="3.069232357s" totalImagesPullingTime="1.625611792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:22.068435992 +0000 UTC m=+88.938483161" watchObservedRunningTime="2026-07-06 15:35:22.069232357 +0000 UTC m=+88.939279524"
Jul 06 15:35:35.975284 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:35.975256    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vl8xx"
Jul 06 15:35:45.008572 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:45.008544    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-state-metrics/0.log"
Jul 06 15:35:45.209252 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:45.209227    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-rbac-proxy-main/0.log"
Jul 06 15:35:45.408322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:45.408213    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-rbac-proxy-self/0.log"
Jul 06 15:35:45.809107 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:45.809082    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-mb8wc_16183bad-7781-4738-a10e-989bcc171f4e/monitoring-plugin/0.log"
Jul 06 15:35:46.608249 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:46.608220    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/init-textfile/0.log"
Jul 06 15:35:46.810986 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:46.810959    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/node-exporter/0.log"
Jul 06 15:35:47.014295 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:35:47.014269    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/kube-rbac-proxy/0.log"
Jul 06 15:38:53.512576 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:38:53.512546    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 15:40:46.689717 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.689684    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cw728"]
Jul 06 15:40:46.691889 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.691873    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.696176 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.696144    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 15:40:46.696311 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.696181    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 15:40:46.696311 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.696262    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fhqpz\""
Jul 06 15:40:46.705776 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.705752    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cw728"]
Jul 06 15:40:46.835768 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.835729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.835946 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.835786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjhg\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-kube-api-access-ghjhg\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.936848 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.936798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjhg\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-kube-api-access-ghjhg\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.937030 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.936889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.946592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.946560    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:46.946748 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:46.946731    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjhg\" (UniqueName: \"kubernetes.io/projected/dc1886e4-7bf8-4012-a73c-190a1973a77b-kube-api-access-ghjhg\") pod \"cert-manager-cainjector-559cdd8c87-cw728\" (UID: \"dc1886e4-7bf8-4012-a73c-190a1973a77b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:47.000798 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:47.000761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728"
Jul 06 15:40:47.137857 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:47.137828    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cw728"]
Jul 06 15:40:47.140920 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:40:47.140890    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc1886e4_7bf8_4012_a73c_190a1973a77b.slice/crio-ac575236e87e253e9324bcad51db7084e0b408f35dc1ac05b52353f6cf3394d0 WatchSource:0}: Error finding container ac575236e87e253e9324bcad51db7084e0b408f35dc1ac05b52353f6cf3394d0: Status 404 returned error can't find the container with id ac575236e87e253e9324bcad51db7084e0b408f35dc1ac05b52353f6cf3394d0
Jul 06 15:40:47.142777 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:47.142757    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:40:48.116062 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:48.116028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728" event={"ID":"dc1886e4-7bf8-4012-a73c-190a1973a77b","Type":"ContainerStarted","Data":"ac575236e87e253e9324bcad51db7084e0b408f35dc1ac05b52353f6cf3394d0"}
Jul 06 15:40:54.139326 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:54.139288    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728" event={"ID":"dc1886e4-7bf8-4012-a73c-190a1973a77b","Type":"ContainerStarted","Data":"9b00c81c85b1b7c7f203c284ad21107796764a7f45aa01c6b450079d0275bef7"}
Jul 06 15:40:54.160455 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:40:54.160408    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cw728" podStartSLOduration=1.788675833 podStartE2EDuration="8.160391815s" totalImagesPullingTime="6.371715982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:40:54.159657794 +0000 UTC m=+421.029704963" watchObservedRunningTime="2026-07-06 15:40:54.160391815 +0000 UTC m=+421.030438983"
Jul 06 15:41:18.868386 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.868296    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"]
Jul 06 15:41:18.870496 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.870475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:18.873648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.873606    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 15:41:18.873648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.873608    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dj8mj\""
Jul 06 15:41:18.873864 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.873847    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 15:41:18.873924 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.873900    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 15:41:18.874120 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.874105    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 15:41:18.892276 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.892243    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"]
Jul 06 15:41:18.964340 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.964305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:18.964340 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.964344    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw5cg\" (UniqueName: \"kubernetes.io/projected/9199c637-72b2-405f-a477-481ace41076e-kube-api-access-pw5cg\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:18.964592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.964364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:18.964592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.964499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:18.964592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:18.964545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065547 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065547 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw5cg\" (UniqueName: \"kubernetes.io/projected/9199c637-72b2-405f-a477-481ace41076e-kube-api-access-pw5cg\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065547 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.065738 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.065707    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.067895 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.067863    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199c637-72b2-405f-a477-481ace41076e-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.067995 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.067926    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.067995 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.067978    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9199c637-72b2-405f-a477-481ace41076e-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.076830 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.076810    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw5cg\" (UniqueName: \"kubernetes.io/projected/9199c637-72b2-405f-a477-481ace41076e-kube-api-access-pw5cg\") pod \"opendatahub-operator-controller-manager-749df7788b-47q8t\" (UID: \"9199c637-72b2-405f-a477-481ace41076e\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.181137 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.181046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:19.333456 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:19.333423    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"]
Jul 06 15:41:19.336853 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:19.336824    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9199c637_72b2_405f_a477_481ace41076e.slice/crio-ed2362e3c7741b0b85a6a0d89be5bed6b8692d914b0c0a6dae131caa1770315e WatchSource:0}: Error finding container ed2362e3c7741b0b85a6a0d89be5bed6b8692d914b0c0a6dae131caa1770315e: Status 404 returned error can't find the container with id ed2362e3c7741b0b85a6a0d89be5bed6b8692d914b0c0a6dae131caa1770315e
Jul 06 15:41:20.229405 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:20.229335    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t" event={"ID":"9199c637-72b2-405f-a477-481ace41076e","Type":"ContainerStarted","Data":"ed2362e3c7741b0b85a6a0d89be5bed6b8692d914b0c0a6dae131caa1770315e"}
Jul 06 15:41:23.242282 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:23.242238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t" event={"ID":"9199c637-72b2-405f-a477-481ace41076e","Type":"ContainerStarted","Data":"6d99630cb24565295b188659597c3d8d8907791b7f4421a672e65f9eebdb287b"}
Jul 06 15:41:24.247155 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:24.247123    2567 generic.go:356] "Generic (PLEG): container finished" podID="9199c637-72b2-405f-a477-481ace41076e" containerID="6d99630cb24565295b188659597c3d8d8907791b7f4421a672e65f9eebdb287b" exitCode=0
Jul 06 15:41:24.247552 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:24.247190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t" event={"ID":"9199c637-72b2-405f-a477-481ace41076e","Type":"ContainerDied","Data":"6d99630cb24565295b188659597c3d8d8907791b7f4421a672e65f9eebdb287b"}
Jul 06 15:41:25.252288 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:25.252251    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t" event={"ID":"9199c637-72b2-405f-a477-481ace41076e","Type":"ContainerStarted","Data":"c1bfe08a5207cba73f4cc33db8ba2feb81eca60c06ff0fa102b6c286fc1645e9"}
Jul 06 15:41:25.252709 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:25.252376    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:25.280127 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:25.280079    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t" podStartSLOduration=2.931201083 podStartE2EDuration="7.280062614s" totalImagesPullingTime="3.309780547s" totalInitContainerRuntime="1.039080984s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:25.279157073 +0000 UTC m=+452.149204245" watchObservedRunningTime="2026-07-06 15:41:25.280062614 +0000 UTC m=+452.150109886"
Jul 06 15:41:36.259078 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:36.259044    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-47q8t"
Jul 06 15:41:39.603859 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.603824    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8g5d5"]
Jul 06 15:41:39.608696 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.608655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.645000 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.644970    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-745c454698-g7b2h"]
Jul 06 15:41:39.647392 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.647371    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.656842 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.656825    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 15:41:39.656946 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.656825    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 15:41:39.656946 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.656858    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bqv6c\""
Jul 06 15:41:39.663822 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.663790    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 15:41:39.663943 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.663854    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 15:41:39.669638 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.669617    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8g5d5"]
Jul 06 15:41:39.675592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.675573    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xtnk9\""
Jul 06 15:41:39.688579 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.688556    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-745c454698-g7b2h"]
Jul 06 15:41:39.695632 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.695610    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 06 15:41:39.727342 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.727309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl5f\" (UniqueName: \"kubernetes.io/projected/40161411-ffd3-4958-b82c-569fa110d209-kube-api-access-cpl5f\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.727468 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.727356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czts\" (UniqueName: \"kubernetes.io/projected/1d8c4500-9880-4dfc-b9bb-9a80833ece28-kube-api-access-2czts\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.727468 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.727422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40161411-ffd3-4958-b82c-569fa110d209-tls-certs\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.727581 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.727484    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.727581 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.727524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40161411-ffd3-4958-b82c-569fa110d209-tmp\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.828271 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.828236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl5f\" (UniqueName: \"kubernetes.io/projected/40161411-ffd3-4958-b82c-569fa110d209-kube-api-access-cpl5f\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.828431 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.828279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czts\" (UniqueName: \"kubernetes.io/projected/1d8c4500-9880-4dfc-b9bb-9a80833ece28-kube-api-access-2czts\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.828431 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.828309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40161411-ffd3-4958-b82c-569fa110d209-tls-certs\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.828431 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.828339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.828431 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.828372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40161411-ffd3-4958-b82c-569fa110d209-tmp\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.828766 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:41:39.828493    2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 06 15:41:39.828766 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:41:39.828565    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert podName:1d8c4500-9880-4dfc-b9bb-9a80833ece28 nodeName:}" failed. No retries permitted until 2026-07-06 15:41:40.328542524 +0000 UTC m=+467.198589671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert") pod "odh-model-controller-7998c67dc4-8g5d5" (UID: "1d8c4500-9880-4dfc-b9bb-9a80833ece28") : secret "odh-model-controller-webhook-cert" not found
Jul 06 15:41:39.830747 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.830722    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40161411-ffd3-4958-b82c-569fa110d209-tmp\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.830902 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.830882    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40161411-ffd3-4958-b82c-569fa110d209-tls-certs\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.872565 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.872497    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl5f\" (UniqueName: \"kubernetes.io/projected/40161411-ffd3-4958-b82c-569fa110d209-kube-api-access-cpl5f\") pod \"kube-auth-proxy-745c454698-g7b2h\" (UID: \"40161411-ffd3-4958-b82c-569fa110d209\") " pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:39.872816 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.872793    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czts\" (UniqueName: \"kubernetes.io/projected/1d8c4500-9880-4dfc-b9bb-9a80833ece28-kube-api-access-2czts\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:39.956990 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:39.956952    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h"
Jul 06 15:41:40.118056 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.118019    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-745c454698-g7b2h"]
Jul 06 15:41:40.121000 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:40.120968    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40161411_ffd3_4958_b82c_569fa110d209.slice/crio-31e5cd2c6af4b7650a4d0d21dc83f90fd6e36a4e04ec0594bdab532ac387bef8 WatchSource:0}: Error finding container 31e5cd2c6af4b7650a4d0d21dc83f90fd6e36a4e04ec0594bdab532ac387bef8: Status 404 returned error can't find the container with id 31e5cd2c6af4b7650a4d0d21dc83f90fd6e36a4e04ec0594bdab532ac387bef8
Jul 06 15:41:40.304158 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.304115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h" event={"ID":"40161411-ffd3-4958-b82c-569fa110d209","Type":"ContainerStarted","Data":"31e5cd2c6af4b7650a4d0d21dc83f90fd6e36a4e04ec0594bdab532ac387bef8"}
Jul 06 15:41:40.331609 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.331569    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:40.334139 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.334117    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8c4500-9880-4dfc-b9bb-9a80833ece28-cert\") pod \"odh-model-controller-7998c67dc4-8g5d5\" (UID: \"1d8c4500-9880-4dfc-b9bb-9a80833ece28\") " pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:40.519120 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.518773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:40.668732 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:40.668702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8g5d5"]
Jul 06 15:41:40.671397 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:40.671366    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8c4500_9880_4dfc_b9bb_9a80833ece28.slice/crio-0f512b27e8065dab58c89542f479cb4f0b39168813264358ad62eb2cf0344400 WatchSource:0}: Error finding container 0f512b27e8065dab58c89542f479cb4f0b39168813264358ad62eb2cf0344400: Status 404 returned error can't find the container with id 0f512b27e8065dab58c89542f479cb4f0b39168813264358ad62eb2cf0344400
Jul 06 15:41:41.309593 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.309548    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5" event={"ID":"1d8c4500-9880-4dfc-b9bb-9a80833ece28","Type":"ContainerStarted","Data":"0f512b27e8065dab58c89542f479cb4f0b39168813264358ad62eb2cf0344400"}
Jul 06 15:41:41.888303 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.888257    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:41.890549 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.890530    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.896303 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.895892    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 06 15:41:41.896303 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.895904    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 06 15:41:41.896303 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.896137    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tgzdx\""
Jul 06 15:41:41.897143 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.897085    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 15:41:41.897923 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.897890    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 06 15:41:41.921733 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.921651    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:41.949356 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949356 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r245g\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949356 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949567    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:41.949627 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:41.949597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.050988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051093    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r245g\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.051425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.051316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.052821 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.052749    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.054121 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.054096    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.054246 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.054142    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.054948 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.054908    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.056022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.055998    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.065005 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.064958    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.066014 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.065971    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r245g\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g\") pod \"istiod-openshift-gateway-7574459656-s8hhc\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:42.203531 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:42.203482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:43.037695 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:43.037588    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:43.041026 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:43.040994    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5490f94_77e1_4eb2_a59b_c22b3bf548ec.slice/crio-2d654ef0b4016ec806ae09746208d3cc6259031209182e197872ba8750b80413 WatchSource:0}: Error finding container 2d654ef0b4016ec806ae09746208d3cc6259031209182e197872ba8750b80413: Status 404 returned error can't find the container with id 2d654ef0b4016ec806ae09746208d3cc6259031209182e197872ba8750b80413
Jul 06 15:41:43.320167 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:43.320133    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" event={"ID":"d5490f94-77e1-4eb2-a59b-c22b3bf548ec","Type":"ContainerStarted","Data":"2d654ef0b4016ec806ae09746208d3cc6259031209182e197872ba8750b80413"}
Jul 06 15:41:45.331140 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:45.331098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5" event={"ID":"1d8c4500-9880-4dfc-b9bb-9a80833ece28","Type":"ContainerStarted","Data":"281c4280f7aa1b36006dd88ab5eaf8cc18cf73203a785f13a2b759c5ed0bebad"}
Jul 06 15:41:45.331596 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:45.331210    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:45.332727 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:45.332695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h" event={"ID":"40161411-ffd3-4958-b82c-569fa110d209","Type":"ContainerStarted","Data":"6d546578310c5d0df576947ca02ecc7f161b6d12833c899b739732ed519bdc75"}
Jul 06 15:41:45.354550 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:45.354494    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5" podStartSLOduration=2.443152079 podStartE2EDuration="6.354475388s" totalImagesPullingTime="3.911323309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:45.352862949 +0000 UTC m=+472.222910117" watchObservedRunningTime="2026-07-06 15:41:45.354475388 +0000 UTC m=+472.224522559"
Jul 06 15:41:45.372835 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:45.372738    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-745c454698-g7b2h" podStartSLOduration=1.9104848479999998 podStartE2EDuration="6.372718238s" totalImagesPullingTime="4.46223339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:45.370805785 +0000 UTC m=+472.240852947" watchObservedRunningTime="2026-07-06 15:41:45.372718238 +0000 UTC m=+472.242765407"
Jul 06 15:41:46.440920 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:46.440860    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:41:46.441208 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:46.440956    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:41:47.342743 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:47.342706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" event={"ID":"d5490f94-77e1-4eb2-a59b-c22b3bf548ec","Type":"ContainerStarted","Data":"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"}
Jul 06 15:41:47.342931 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:47.342913    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:47.344585 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:47.344558    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-s8hhc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:41:47.344729 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:47.344605    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:41:47.491575 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:47.491518    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" podStartSLOduration=3.094314942 podStartE2EDuration="6.491501348s" totalImagesPullingTime="3.397186406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:47.486839307 +0000 UTC m=+474.356886476" watchObservedRunningTime="2026-07-06 15:41:47.491501348 +0000 UTC m=+474.361548515"
Jul 06 15:41:48.346861 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:48.346816    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-s8hhc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:41:48.347031 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:48.346890    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:41:49.034141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.034102    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"]
Jul 06 15:41:49.037296 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.037279    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.048263 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.048241    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zpjkn\""
Jul 06 15:41:49.078279 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.078247    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"]
Jul 06 15:41:49.112148 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx6k\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-kube-api-access-jkx6k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112162    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112438 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112295    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112438 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112438 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112350    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.112438 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.112382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213733 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx6k\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-kube-api-access-jkx6k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213765    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.213883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214186 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214186 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.213938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214286 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.214235    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214343 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.214301    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214423 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.214393    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214493 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.214472    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.214685 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.214639    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.216276 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.216245    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.216541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.216524    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.229519 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.229497    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx6k\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-kube-api-access-jkx6k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.234570 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.234552    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47c5e9-014c-45a1-a6aa-506dbf588bd9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tcldv\" (UID: \"4b47c5e9-014c-45a1-a6aa-506dbf588bd9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.347321 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.347228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:49.536597 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:49.536170    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"]
Jul 06 15:41:49.540627 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:49.540587    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b47c5e9_014c_45a1_a6aa_506dbf588bd9.slice/crio-6cb932af4eca806a0b67577ac6740628d6179ad12f016d166458eadd150b3f91 WatchSource:0}: Error finding container 6cb932af4eca806a0b67577ac6740628d6179ad12f016d166458eadd150b3f91: Status 404 returned error can't find the container with id 6cb932af4eca806a0b67577ac6740628d6179ad12f016d166458eadd150b3f91
Jul 06 15:41:50.357235 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:50.357189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv" event={"ID":"4b47c5e9-014c-45a1-a6aa-506dbf588bd9","Type":"ContainerStarted","Data":"6cb932af4eca806a0b67577ac6740628d6179ad12f016d166458eadd150b3f91"}
Jul 06 15:41:51.347707 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:51.347653    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:52.128175 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.128116    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gk459"]
Jul 06 15:41:52.131083 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.131062    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.134600 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.134579    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 06 15:41:52.134809 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.134740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f8cd5\""
Jul 06 15:41:52.135622 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.135599    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"]
Jul 06 15:41:52.139003 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.138631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.141731 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.141708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 06 15:41:52.141857 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.141717    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j7xcf\""
Jul 06 15:41:52.144594 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.144572    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gk459"]
Jul 06 15:41:52.154141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.154109    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:41:52.154229 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.154172    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:41:52.154229 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.154198    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:41:52.157231 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.157211    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"]
Jul 06 15:41:52.241564 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.241541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.241723 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.241598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af73161c-e2d8-4380-966b-7f04591726e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.241723 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.241650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cwf\" (UniqueName: \"kubernetes.io/projected/af73161c-e2d8-4380-966b-7f04591726e6-kube-api-access-b6cwf\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.241815 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.241740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghvf\" (UniqueName: \"kubernetes.io/projected/e57cc142-4bec-42eb-8a44-511b68d2b09e-kube-api-access-rghvf\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.342917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.342871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af73161c-e2d8-4380-966b-7f04591726e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.342917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.342916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cwf\" (UniqueName: \"kubernetes.io/projected/af73161c-e2d8-4380-966b-7f04591726e6-kube-api-access-b6cwf\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.343176 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.342960    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghvf\" (UniqueName: \"kubernetes.io/projected/e57cc142-4bec-42eb-8a44-511b68d2b09e-kube-api-access-rghvf\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.343176 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.343137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.343279 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:41:52.343250    2567 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 06 15:41:52.343331 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:41:52.343318    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert podName:e57cc142-4bec-42eb-8a44-511b68d2b09e nodeName:}" failed. No retries permitted until 2026-07-06 15:41:52.843284868 +0000 UTC m=+479.713332014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert") pod "llmisvc-controller-manager-777c6bd7d5-9lllh" (UID: "e57cc142-4bec-42eb-8a44-511b68d2b09e") : secret "llmisvc-webhook-server-cert" not found
Jul 06 15:41:52.345528 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.345504    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af73161c-e2d8-4380-966b-7f04591726e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.360687 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.360639    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cwf\" (UniqueName: \"kubernetes.io/projected/af73161c-e2d8-4380-966b-7f04591726e6-kube-api-access-b6cwf\") pod \"kserve-controller-manager-d6f4b65b8-gk459\" (UID: \"af73161c-e2d8-4380-966b-7f04591726e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.361032 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.361011    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghvf\" (UniqueName: \"kubernetes.io/projected/e57cc142-4bec-42eb-8a44-511b68d2b09e-kube-api-access-rghvf\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.370742 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.369911    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv" event={"ID":"4b47c5e9-014c-45a1-a6aa-506dbf588bd9","Type":"ContainerStarted","Data":"352f3152430f054a363207a17d3aaf82df2d674fc57821866e4f79481dfebe73"}
Jul 06 15:41:52.415439 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.415331    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv" podStartSLOduration=1.803977493 podStartE2EDuration="4.415312648s" totalImagesPullingTime="2.611335155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:52.41408662 +0000 UTC m=+479.284133789" watchObservedRunningTime="2026-07-06 15:41:52.415312648 +0000 UTC m=+479.285359815"
Jul 06 15:41:52.443270 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.443236    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:52.639254 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.639222    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gk459"]
Jul 06 15:41:52.641152 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:52.641119    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf73161c_e2d8_4380_966b_7f04591726e6.slice/crio-598258d0a439427fda5d45c0bde83e11490644f18b37b21c5e029a5887d216ff WatchSource:0}: Error finding container 598258d0a439427fda5d45c0bde83e11490644f18b37b21c5e029a5887d216ff: Status 404 returned error can't find the container with id 598258d0a439427fda5d45c0bde83e11490644f18b37b21c5e029a5887d216ff
Jul 06 15:41:52.849543 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.849490    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:52.852066 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:52.852042    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e57cc142-4bec-42eb-8a44-511b68d2b09e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9lllh\" (UID: \"e57cc142-4bec-42eb-8a44-511b68d2b09e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:53.050557 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.050521    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:53.188252 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.188224    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"]
Jul 06 15:41:53.189505 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:41:53.189481    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode57cc142_4bec_42eb_8a44_511b68d2b09e.slice/crio-28e0ff91779bb408b6c147c704600da78a4e77297816e21e58fa77ed915fcdfc WatchSource:0}: Error finding container 28e0ff91779bb408b6c147c704600da78a4e77297816e21e58fa77ed915fcdfc: Status 404 returned error can't find the container with id 28e0ff91779bb408b6c147c704600da78a4e77297816e21e58fa77ed915fcdfc
Jul 06 15:41:53.348259 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.348219    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:53.353389 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.353324    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:53.376639 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.376604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh" event={"ID":"e57cc142-4bec-42eb-8a44-511b68d2b09e","Type":"ContainerStarted","Data":"28e0ff91779bb408b6c147c704600da78a4e77297816e21e58fa77ed915fcdfc"}
Jul 06 15:41:53.378021 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.377982    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459" event={"ID":"af73161c-e2d8-4380-966b-7f04591726e6","Type":"ContainerStarted","Data":"598258d0a439427fda5d45c0bde83e11490644f18b37b21c5e029a5887d216ff"}
Jul 06 15:41:53.378341 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.378323    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:53.379494 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:53.379473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tcldv"
Jul 06 15:41:56.338785 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:56.338752    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8g5d5"
Jul 06 15:41:56.663389 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:56.663301    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:56.663734 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:56.663570    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery" containerID="cri-o://7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf" gracePeriod=30
Jul 06 15:41:57.338993 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.338963    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:57.399169 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.399131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh" event={"ID":"e57cc142-4bec-42eb-8a44-511b68d2b09e","Type":"ContainerStarted","Data":"595f32fee2b103e919be0d45b24013610d0918386316c2918da2e692d68d7765"}
Jul 06 15:41:57.399342 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.399266    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:41:57.400770 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.400738    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459" event={"ID":"af73161c-e2d8-4380-966b-7f04591726e6","Type":"ContainerStarted","Data":"246d773162c223a983a9ad77978bd2710446f0bd81b8464d5d588e3062b2c0ab"}
Jul 06 15:41:57.400898 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.400800    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:41:57.401992 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.401966    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerID="7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf" exitCode=0
Jul 06 15:41:57.402076 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.402004    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"
Jul 06 15:41:57.402125 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.402038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" event={"ID":"d5490f94-77e1-4eb2-a59b-c22b3bf548ec","Type":"ContainerDied","Data":"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"}
Jul 06 15:41:57.402125 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.402112    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc" event={"ID":"d5490f94-77e1-4eb2-a59b-c22b3bf548ec","Type":"ContainerDied","Data":"2d654ef0b4016ec806ae09746208d3cc6259031209182e197872ba8750b80413"}
Jul 06 15:41:57.402196 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.402144    2567 scope.go:117] "RemoveContainer" containerID="7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"
Jul 06 15:41:57.410806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.410790    2567 scope.go:117] "RemoveContainer" containerID="7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"
Jul 06 15:41:57.411079 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:41:57.411059    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf\": container with ID starting with 7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf not found: ID does not exist" containerID="7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"
Jul 06 15:41:57.411141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.411088    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf"} err="failed to get container status \"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf\": rpc error: code = NotFound desc = could not find container \"7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf\": container with ID starting with 7f743a03e80ebe36f5b87ec4b878547c14b4e7256d2e64f749417c1be0777cbf not found: ID does not exist"
Jul 06 15:41:57.421814 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.421765    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh" podStartSLOduration=1.3956312180000001 podStartE2EDuration="5.421749473s" totalImagesPullingTime="4.026118255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:57.419543029 +0000 UTC m=+484.289590196" watchObservedRunningTime="2026-07-06 15:41:57.421749473 +0000 UTC m=+484.291796641"
Jul 06 15:41:57.442896 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.442820    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459" podStartSLOduration=0.873588397 podStartE2EDuration="5.44280162s" totalImagesPullingTime="4.569213223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:57.442302901 +0000 UTC m=+484.312350071" watchObservedRunningTime="2026-07-06 15:41:57.44280162 +0000 UTC m=+484.312848791"
Jul 06 15:41:57.494529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494435    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494490    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494516    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494786 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494630    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494786 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494719    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494797    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r245g\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.494891 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.494824    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts\") pod \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\" (UID: \"d5490f94-77e1-4eb2-a59b-c22b3bf548ec\") "
Jul 06 15:41:57.495390 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.495257    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:41:57.497745 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.497708    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:41:57.497745 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.497719    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs" (OuterVolumeSpecName: "local-certs") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:41:57.497900 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.497799    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:41:57.497900 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.497827    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts" (OuterVolumeSpecName: "cacerts") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:41:57.497999 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.497902    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:41:57.498219 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.498204    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g" (OuterVolumeSpecName: "kube-api-access-r245g") pod "d5490f94-77e1-4eb2-a59b-c22b3bf548ec" (UID: "d5490f94-77e1-4eb2-a59b-c22b3bf548ec"). InnerVolumeSpecName "kube-api-access-r245g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:41:57.596133 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596092    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r245g\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-kube-api-access-r245g\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596133 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596131    2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-cacerts\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596146    2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-kubeconfig\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596161    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-dns-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596174    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-csr-ca-configmap\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596187    2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-local-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.596330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.596199    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5490f94-77e1-4eb2-a59b-c22b3bf548ec-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:41:57.718387 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.718358    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:57.722211 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:57.722183    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-s8hhc"]
Jul 06 15:41:59.663359 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:41:59.663310    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" path="/var/lib/kubelet/pods/d5490f94-77e1-4eb2-a59b-c22b3bf548ec/volumes"
Jul 06 15:42:28.409217 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:28.409187    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9lllh"
Jul 06 15:42:28.412268 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:28.412241    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gk459"
Jul 06 15:42:40.303268 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.303233    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:40.303648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.303520    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery"
Jul 06 15:42:40.303648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.303530    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery"
Jul 06 15:42:40.303648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.303621    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5490f94-77e1-4eb2-a59b-c22b3bf548ec" containerName="discovery"
Jul 06 15:42:40.314450 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.314416    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:40.316940 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.316914    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 15:42:40.317117 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.316954    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 15:42:40.317431 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.317407    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:40.317854 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.317832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-88t4l\""
Jul 06 15:42:40.436523 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.436493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f4r\" (UniqueName: \"kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r\") pod \"kuadrant-operator-catalog-pvwq2\" (UID: \"34822239-aff6-4b8c-b576-c519bc1ed739\") " pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:40.537751 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.537713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f4r\" (UniqueName: \"kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r\") pod \"kuadrant-operator-catalog-pvwq2\" (UID: \"34822239-aff6-4b8c-b576-c519bc1ed739\") " pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:40.546571 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.546534    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f4r\" (UniqueName: \"kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r\") pod \"kuadrant-operator-catalog-pvwq2\" (UID: \"34822239-aff6-4b8c-b576-c519bc1ed739\") " pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:40.626489 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.626389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:40.666584 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.666555    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:40.755103 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.755051    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:40.758839 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:42:40.758814    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34822239_aff6_4b8c_b576_c519bc1ed739.slice/crio-13c169a40349960ec33eeeec2867cc9dfe4f8abae998cef71af7049bd5589ab1 WatchSource:0}: Error finding container 13c169a40349960ec33eeeec2867cc9dfe4f8abae998cef71af7049bd5589ab1: Status 404 returned error can't find the container with id 13c169a40349960ec33eeeec2867cc9dfe4f8abae998cef71af7049bd5589ab1
Jul 06 15:42:40.883516 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.883443    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8cb9d"]
Jul 06 15:42:40.886113 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.886096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:40.896443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.896419    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8cb9d"]
Jul 06 15:42:40.940388 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:40.940358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwr2\" (UniqueName: \"kubernetes.io/projected/2642697c-4919-4fb3-9b50-69dbdf2d40d9-kube-api-access-6xwr2\") pod \"kuadrant-operator-catalog-8cb9d\" (UID: \"2642697c-4919-4fb3-9b50-69dbdf2d40d9\") " pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:41.040895 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.040856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwr2\" (UniqueName: \"kubernetes.io/projected/2642697c-4919-4fb3-9b50-69dbdf2d40d9-kube-api-access-6xwr2\") pod \"kuadrant-operator-catalog-8cb9d\" (UID: \"2642697c-4919-4fb3-9b50-69dbdf2d40d9\") " pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:41.051060 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.051032    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwr2\" (UniqueName: \"kubernetes.io/projected/2642697c-4919-4fb3-9b50-69dbdf2d40d9-kube-api-access-6xwr2\") pod \"kuadrant-operator-catalog-8cb9d\" (UID: \"2642697c-4919-4fb3-9b50-69dbdf2d40d9\") " pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:41.195160 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.195122    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:41.328158 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.328119    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8cb9d"]
Jul 06 15:42:41.366493 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:42:41.366458    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2642697c_4919_4fb3_9b50_69dbdf2d40d9.slice/crio-1054800ec46d5a066b04fd0fb1fa15027b81ec2a08ff79352386b6401a10c914 WatchSource:0}: Error finding container 1054800ec46d5a066b04fd0fb1fa15027b81ec2a08ff79352386b6401a10c914: Status 404 returned error can't find the container with id 1054800ec46d5a066b04fd0fb1fa15027b81ec2a08ff79352386b6401a10c914
Jul 06 15:42:41.584459 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.584131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d" event={"ID":"2642697c-4919-4fb3-9b50-69dbdf2d40d9","Type":"ContainerStarted","Data":"1054800ec46d5a066b04fd0fb1fa15027b81ec2a08ff79352386b6401a10c914"}
Jul 06 15:42:41.587727 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:41.586097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" event={"ID":"34822239-aff6-4b8c-b576-c519bc1ed739","Type":"ContainerStarted","Data":"13c169a40349960ec33eeeec2867cc9dfe4f8abae998cef71af7049bd5589ab1"}
Jul 06 15:42:43.596403 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.596363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" event={"ID":"34822239-aff6-4b8c-b576-c519bc1ed739","Type":"ContainerStarted","Data":"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"}
Jul 06 15:42:43.596901 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.596422    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" podUID="34822239-aff6-4b8c-b576-c519bc1ed739" containerName="registry-server" containerID="cri-o://a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459" gracePeriod=2
Jul 06 15:42:43.597940 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.597914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d" event={"ID":"2642697c-4919-4fb3-9b50-69dbdf2d40d9","Type":"ContainerStarted","Data":"f672b14c626934651dc52e19f41b4436195ecf5e148928f4b4c1342291f1c99c"}
Jul 06 15:42:43.613774 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.613739    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" podStartSLOduration=1.673871567 podStartE2EDuration="3.613725178s" totalImagesPullingTime="1.939853611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:42:43.61305949 +0000 UTC m=+530.483106659" watchObservedRunningTime="2026-07-06 15:42:43.613725178 +0000 UTC m=+530.483772346"
Jul 06 15:42:43.631758 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.631717    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d" podStartSLOduration=2.296366514 podStartE2EDuration="3.631703893s" totalImagesPullingTime="1.335337379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:42:43.630629613 +0000 UTC m=+530.500676780" watchObservedRunningTime="2026-07-06 15:42:43.631703893 +0000 UTC m=+530.501751061"
Jul 06 15:42:43.835589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.835567    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:43.863961 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.863888    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5f4r\" (UniqueName: \"kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r\") pod \"34822239-aff6-4b8c-b576-c519bc1ed739\" (UID: \"34822239-aff6-4b8c-b576-c519bc1ed739\") "
Jul 06 15:42:43.866072 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.866051    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r" (OuterVolumeSpecName: "kube-api-access-b5f4r") pod "34822239-aff6-4b8c-b576-c519bc1ed739" (UID: "34822239-aff6-4b8c-b576-c519bc1ed739"). InnerVolumeSpecName "kube-api-access-b5f4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:42:43.964486 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:43.964456    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5f4r\" (UniqueName: \"kubernetes.io/projected/34822239-aff6-4b8c-b576-c519bc1ed739-kube-api-access-b5f4r\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:42:44.603306 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.603267    2567 generic.go:356] "Generic (PLEG): container finished" podID="34822239-aff6-4b8c-b576-c519bc1ed739" containerID="a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459" exitCode=0
Jul 06 15:42:44.603831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.603308    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2"
Jul 06 15:42:44.603831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.603354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" event={"ID":"34822239-aff6-4b8c-b576-c519bc1ed739","Type":"ContainerDied","Data":"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"}
Jul 06 15:42:44.603831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.603411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvwq2" event={"ID":"34822239-aff6-4b8c-b576-c519bc1ed739","Type":"ContainerDied","Data":"13c169a40349960ec33eeeec2867cc9dfe4f8abae998cef71af7049bd5589ab1"}
Jul 06 15:42:44.603831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.603432    2567 scope.go:117] "RemoveContainer" containerID="a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"
Jul 06 15:42:44.612100 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.612083    2567 scope.go:117] "RemoveContainer" containerID="a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"
Jul 06 15:42:44.612351 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:42:44.612331    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459\": container with ID starting with a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459 not found: ID does not exist" containerID="a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"
Jul 06 15:42:44.612396 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.612360    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459"} err="failed to get container status \"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459\": rpc error: code = NotFound desc = could not find container \"a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459\": container with ID starting with a4384892fad2db28491e1dacfcc18a9c4848fcc84f97c2f0d1f4c7126eac0459 not found: ID does not exist"
Jul 06 15:42:44.624514 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.624492    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:44.631330 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:44.631312    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvwq2"]
Jul 06 15:42:45.663300 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:45.663267    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34822239-aff6-4b8c-b576-c519bc1ed739" path="/var/lib/kubelet/pods/34822239-aff6-4b8c-b576-c519bc1ed739/volumes"
Jul 06 15:42:51.195917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:51.195870    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:51.196321 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:51.195940    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:51.218549 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:51.218524    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:42:51.654821 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:42:51.654738    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8cb9d"
Jul 06 15:43:11.985429 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.985394    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"]
Jul 06 15:43:11.985963 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.985708    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34822239-aff6-4b8c-b576-c519bc1ed739" containerName="registry-server"
Jul 06 15:43:11.985963 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.985721    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="34822239-aff6-4b8c-b576-c519bc1ed739" containerName="registry-server"
Jul 06 15:43:11.985963 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.985791    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="34822239-aff6-4b8c-b576-c519bc1ed739" containerName="registry-server"
Jul 06 15:43:11.988719 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.988690    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:11.991044 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.991020    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p6ld8\""
Jul 06 15:43:12.000421 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:11.998573    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"]
Jul 06 15:43:12.095181 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.095148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7s7w\" (UniqueName: \"kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w\") pod \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" (UID: \"8d6a28b8-d5d2-486a-9259-986e1854dccf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:12.196559 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.196521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7s7w\" (UniqueName: \"kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w\") pod \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" (UID: \"8d6a28b8-d5d2-486a-9259-986e1854dccf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:12.211213 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.211178    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7s7w\" (UniqueName: \"kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w\") pod \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" (UID: \"8d6a28b8-d5d2-486a-9259-986e1854dccf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:12.304129 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.304023    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:12.442097 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.442042    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"]
Jul 06 15:43:12.444260 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:12.444231    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6a28b8_d5d2_486a_9259_986e1854dccf.slice/crio-1c695d2751fb51978aadd04deba22aa5af04a1112eda709a22be0fc801efc4c7 WatchSource:0}: Error finding container 1c695d2751fb51978aadd04deba22aa5af04a1112eda709a22be0fc801efc4c7: Status 404 returned error can't find the container with id 1c695d2751fb51978aadd04deba22aa5af04a1112eda709a22be0fc801efc4c7
Jul 06 15:43:12.716331 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:12.716296    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg" event={"ID":"8d6a28b8-d5d2-486a-9259-986e1854dccf","Type":"ContainerStarted","Data":"1c695d2751fb51978aadd04deba22aa5af04a1112eda709a22be0fc801efc4c7"}
Jul 06 15:43:14.727264 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:14.727229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg" event={"ID":"8d6a28b8-d5d2-486a-9259-986e1854dccf","Type":"ContainerStarted","Data":"f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c"}
Jul 06 15:43:14.727688 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:14.727309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:14.747895 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:14.747847    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg" podStartSLOduration=1.8328658070000001 podStartE2EDuration="3.747831964s" totalImagesPullingTime="1.914966157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:14.746451273 +0000 UTC m=+561.616498440" watchObservedRunningTime="2026-07-06 15:43:14.747831964 +0000 UTC m=+561.617879169"
Jul 06 15:43:16.066622 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.066586    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"]
Jul 06 15:43:16.069869 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.069849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:16.072478 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.072458    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8ngw4\""
Jul 06 15:43:16.083561 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.083537    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"]
Jul 06 15:43:16.231750 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.231717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sflh\" (UniqueName: \"kubernetes.io/projected/0b0a5ad8-6685-442a-993b-0af67a573139-kube-api-access-9sflh\") pod \"authorino-operator-75cc8d5cb7-gbhqq\" (UID: \"0b0a5ad8-6685-442a-993b-0af67a573139\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:16.333221 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.333128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sflh\" (UniqueName: \"kubernetes.io/projected/0b0a5ad8-6685-442a-993b-0af67a573139-kube-api-access-9sflh\") pod \"authorino-operator-75cc8d5cb7-gbhqq\" (UID: \"0b0a5ad8-6685-442a-993b-0af67a573139\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:16.350052 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.350012    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sflh\" (UniqueName: \"kubernetes.io/projected/0b0a5ad8-6685-442a-993b-0af67a573139-kube-api-access-9sflh\") pod \"authorino-operator-75cc8d5cb7-gbhqq\" (UID: \"0b0a5ad8-6685-442a-993b-0af67a573139\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:16.379795 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.379756    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:16.557123 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.557092    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"]
Jul 06 15:43:16.559289 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:16.559262    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0a5ad8_6685_442a_993b_0af67a573139.slice/crio-2ebceeda451783ade84ffda9d5cfe1a60df970cad8c62cd837c8e02d5c5a28c0 WatchSource:0}: Error finding container 2ebceeda451783ade84ffda9d5cfe1a60df970cad8c62cd837c8e02d5c5a28c0: Status 404 returned error can't find the container with id 2ebceeda451783ade84ffda9d5cfe1a60df970cad8c62cd837c8e02d5c5a28c0
Jul 06 15:43:16.742144 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:16.742107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq" event={"ID":"0b0a5ad8-6685-442a-993b-0af67a573139","Type":"ContainerStarted","Data":"2ebceeda451783ade84ffda9d5cfe1a60df970cad8c62cd837c8e02d5c5a28c0"}
Jul 06 15:43:19.757825 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:19.757791    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq" event={"ID":"0b0a5ad8-6685-442a-993b-0af67a573139","Type":"ContainerStarted","Data":"e00146abe3be2867e5d3d9183781df27890dcea21eb6b7c0d5f31bf743167652"}
Jul 06 15:43:19.758312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:19.757878    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:19.779291 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:19.779231    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq" podStartSLOduration=1.580697075 podStartE2EDuration="3.77921127s" totalImagesPullingTime="2.198514195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:19.778406455 +0000 UTC m=+566.648453623" watchObservedRunningTime="2026-07-06 15:43:19.77921127 +0000 UTC m=+566.649258439"
Jul 06 15:43:21.175570 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.175534    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"]
Jul 06 15:43:21.177831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.177811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.180367 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.180341    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f6w8p\""
Jul 06 15:43:21.190591 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.190566    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"]
Jul 06 15:43:21.277049 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.277014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.277219 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.277071    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbtn\" (UniqueName: \"kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.378434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.378393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.378611 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.378468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbtn\" (UniqueName: \"kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.378824 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.378806    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.391222 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.391198    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbtn\" (UniqueName: \"kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.487100 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.487071    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:21.619865 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.619839    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"]
Jul 06 15:43:21.622123 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:21.622096    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e699ab7_8bf1_4fba_b842_5bc488af6df5.slice/crio-bf74a137e371a63f794a7cbe2cd5a522ddca1c5c4ef06f47f4667b67645d5a1d WatchSource:0}: Error finding container bf74a137e371a63f794a7cbe2cd5a522ddca1c5c4ef06f47f4667b67645d5a1d: Status 404 returned error can't find the container with id bf74a137e371a63f794a7cbe2cd5a522ddca1c5c4ef06f47f4667b67645d5a1d
Jul 06 15:43:21.768372 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:21.768283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6" event={"ID":"9e699ab7-8bf1-4fba-b842-5bc488af6df5","Type":"ContainerStarted","Data":"bf74a137e371a63f794a7cbe2cd5a522ddca1c5c4ef06f47f4667b67645d5a1d"}
Jul 06 15:43:25.734652 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:25.734617    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:28.799711 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:28.799656    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6" event={"ID":"9e699ab7-8bf1-4fba-b842-5bc488af6df5","Type":"ContainerStarted","Data":"9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730"}
Jul 06 15:43:28.800083 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:28.799933    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:28.821241 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:28.821191    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6" podStartSLOduration=1.194050485 podStartE2EDuration="7.821174802s" totalImagesPullingTime="6.627124317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:28.819396741 +0000 UTC m=+575.689443920" watchObservedRunningTime="2026-07-06 15:43:28.821174802 +0000 UTC m=+575.691221960"
Jul 06 15:43:30.765450 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.765420    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gbhqq"
Jul 06 15:43:30.920685 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.920626    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"]
Jul 06 15:43:30.922883 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.922859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:30.925140 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.925118    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 06 15:43:30.925815 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.925799    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 06 15:43:30.925986 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.925969    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jk25k\""
Jul 06 15:43:30.935960 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.935937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"]
Jul 06 15:43:30.962526 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.962498    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:30.962645 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.962532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6587014d-7185-484b-a6e8-15a29fb2b75f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:30.962645 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:30.962566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8lt\" (UniqueName: \"kubernetes.io/projected/6587014d-7185-484b-a6e8-15a29fb2b75f-kube-api-access-zl8lt\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.064054 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.063964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8lt\" (UniqueName: \"kubernetes.io/projected/6587014d-7185-484b-a6e8-15a29fb2b75f-kube-api-access-zl8lt\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.064054 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.064051    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.064242 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.064077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6587014d-7185-484b-a6e8-15a29fb2b75f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.064242 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:31.064206    2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 06 15:43:31.064316 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:31.064265    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert podName:6587014d-7185-484b-a6e8-15a29fb2b75f nodeName:}" failed. No retries permitted until 2026-07-06 15:43:31.564249397 +0000 UTC m=+578.434296542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-llrkc" (UID: "6587014d-7185-484b-a6e8-15a29fb2b75f") : secret "plugin-serving-cert" not found
Jul 06 15:43:31.064728 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.064713    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6587014d-7185-484b-a6e8-15a29fb2b75f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.074222 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.074198    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8lt\" (UniqueName: \"kubernetes.io/projected/6587014d-7185-484b-a6e8-15a29fb2b75f-kube-api-access-zl8lt\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.569201 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.569166    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.571744 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.571725    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6587014d-7185-484b-a6e8-15a29fb2b75f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-llrkc\" (UID: \"6587014d-7185-484b-a6e8-15a29fb2b75f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.832158 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.832070    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"
Jul 06 15:43:31.965917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:31.965878    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc"]
Jul 06 15:43:31.968840 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:31.968808    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6587014d_7185_484b_a6e8_15a29fb2b75f.slice/crio-1f0a09afc653a64c6eb1f822180bb50fecc2b343e42b50dea209f522c0710f25 WatchSource:0}: Error finding container 1f0a09afc653a64c6eb1f822180bb50fecc2b343e42b50dea209f522c0710f25: Status 404 returned error can't find the container with id 1f0a09afc653a64c6eb1f822180bb50fecc2b343e42b50dea209f522c0710f25
Jul 06 15:43:32.818323 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:32.818287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc" event={"ID":"6587014d-7185-484b-a6e8-15a29fb2b75f","Type":"ContainerStarted","Data":"1f0a09afc653a64c6eb1f822180bb50fecc2b343e42b50dea209f522c0710f25"}
Jul 06 15:43:39.806581 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:39.806546    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:41.712470 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.712360    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"]
Jul 06 15:43:41.712974 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.712604    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" containerName="manager" containerID="cri-o://9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730" gracePeriod=2
Jul 06 15:43:41.727655 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.727625    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"]
Jul 06 15:43:41.799843 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.799797    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"]
Jul 06 15:43:41.800154 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.800061    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" containerName="manager" containerID="cri-o://f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c" gracePeriod=2
Jul 06 15:43:41.802034 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.802007    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:43:41.802509 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.802487    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" containerName="manager"
Jul 06 15:43:41.802606 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.802511    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" containerName="manager"
Jul 06 15:43:41.802677 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.802614    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" containerName="manager"
Jul 06 15:43:41.805556 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:41.805514    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:41.809186 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.809166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.825831 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.825805    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"]
Jul 06 15:43:41.846541 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.846334    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"]
Jul 06 15:43:41.846769 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.846752    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" containerName="manager"
Jul 06 15:43:41.846863 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.846773    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" containerName="manager"
Jul 06 15:43:41.846922 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.846863    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" containerName="manager"
Jul 06 15:43:41.851313 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.851288    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:41.862565 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.862499    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:43:41.862956 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.862779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6rg\" (UniqueName: \"kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.862956 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.862833    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.879141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.879111    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"]
Jul 06 15:43:41.909624 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:41.909587    2567 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8d6a28b8-d5d2-486a-9259-986e1854dccf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-g8bcg\": pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:41.944958 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:41.944910    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:41.946837 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:41.946810    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:41.963428 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.963334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4vs\" (UniqueName: \"kubernetes.io/projected/7d67df61-0e35-43f7-9edc-93bcbc9c49f5-kube-api-access-9k4vs\") pod \"limitador-operator-controller-manager-859d7b78bd-5wnfn\" (UID: \"7d67df61-0e35-43f7-9edc-93bcbc9c49f5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:41.963573 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.963463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6rg\" (UniqueName: \"kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.963573 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.963510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.963901 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.963883    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:41.980251 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:41.980224    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6rg\" (UniqueName: \"kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6dp6c\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:42.064691 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:42.064645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4vs\" (UniqueName: \"kubernetes.io/projected/7d67df61-0e35-43f7-9edc-93bcbc9c49f5-kube-api-access-9k4vs\") pod \"limitador-operator-controller-manager-859d7b78bd-5wnfn\" (UID: \"7d67df61-0e35-43f7-9edc-93bcbc9c49f5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:42.077947 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:42.077914    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4vs\" (UniqueName: \"kubernetes.io/projected/7d67df61-0e35-43f7-9edc-93bcbc9c49f5-kube-api-access-9k4vs\") pod \"limitador-operator-controller-manager-859d7b78bd-5wnfn\" (UID: \"7d67df61-0e35-43f7-9edc-93bcbc9c49f5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:42.162783 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:42.162746    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:42.170237 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:42.170211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:43.664183 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:43.664148    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:43.665967 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:43.665937    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.299870 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.299840    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.302183 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.302140    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.303403 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.303381    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:54.310779 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.310746    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.315810 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.315774    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:54.330391 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.330368    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:43:54.333041 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:54.333014    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b302d83_cd01_4fde_8218_fe115256ea00.slice/crio-35cbbeae62a232996c3305ddbb31093d2411fd360acd20b0f22d5db16f124f37 WatchSource:0}: Error finding container 35cbbeae62a232996c3305ddbb31093d2411fd360acd20b0f22d5db16f124f37: Status 404 returned error can't find the container with id 35cbbeae62a232996c3305ddbb31093d2411fd360acd20b0f22d5db16f124f37
Jul 06 15:43:54.346511 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.346486    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"]
Jul 06 15:43:54.347085 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:43:54.347061    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d67df61_0e35_43f7_9edc_93bcbc9c49f5.slice/crio-83cad9a8c33d5eda32b507ad4e8bad53d2de0ef89f9b731cd3b69ae1fc8bd5a6 WatchSource:0}: Error finding container 83cad9a8c33d5eda32b507ad4e8bad53d2de0ef89f9b731cd3b69ae1fc8bd5a6: Status 404 returned error can't find the container with id 83cad9a8c33d5eda32b507ad4e8bad53d2de0ef89f9b731cd3b69ae1fc8bd5a6
Jul 06 15:43:54.378592 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.378572    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csbtn\" (UniqueName: \"kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn\") pod \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") "
Jul 06 15:43:54.378734 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.378609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7s7w\" (UniqueName: \"kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w\") pod \"8d6a28b8-d5d2-486a-9259-986e1854dccf\" (UID: \"8d6a28b8-d5d2-486a-9259-986e1854dccf\") "
Jul 06 15:43:54.378734 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.378690    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume\") pod \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\" (UID: \"9e699ab7-8bf1-4fba-b842-5bc488af6df5\") "
Jul 06 15:43:54.379222 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.379196    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e699ab7-8bf1-4fba-b842-5bc488af6df5" (UID: "9e699ab7-8bf1-4fba-b842-5bc488af6df5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:43:54.380707 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.380616    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn" (OuterVolumeSpecName: "kube-api-access-csbtn") pod "9e699ab7-8bf1-4fba-b842-5bc488af6df5" (UID: "9e699ab7-8bf1-4fba-b842-5bc488af6df5"). InnerVolumeSpecName "kube-api-access-csbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:43:54.380780 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.380738    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w" (OuterVolumeSpecName: "kube-api-access-l7s7w") pod "8d6a28b8-d5d2-486a-9259-986e1854dccf" (UID: "8d6a28b8-d5d2-486a-9259-986e1854dccf"). InnerVolumeSpecName "kube-api-access-l7s7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:43:54.480082 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.480031    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbtn\" (UniqueName: \"kubernetes.io/projected/9e699ab7-8bf1-4fba-b842-5bc488af6df5-kube-api-access-csbtn\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:43:54.480082 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.480071    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7s7w\" (UniqueName: \"kubernetes.io/projected/8d6a28b8-d5d2-486a-9259-986e1854dccf-kube-api-access-l7s7w\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:43:54.480082 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.480089    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e699ab7-8bf1-4fba-b842-5bc488af6df5-extensions-socket-volume\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:43:54.934647 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.934612    2567 generic.go:356] "Generic (PLEG): container finished" podID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" containerID="9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730" exitCode=0
Jul 06 15:43:54.934839 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.934652    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.934839 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.934712    2567 scope.go:117] "RemoveContainer" containerID="9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730"
Jul 06 15:43:54.936543 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.936516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" event={"ID":"1b302d83-cd01-4fde-8218-fe115256ea00","Type":"ContainerStarted","Data":"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"}
Jul 06 15:43:54.936709 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.936569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" event={"ID":"1b302d83-cd01-4fde-8218-fe115256ea00","Type":"ContainerStarted","Data":"35cbbeae62a232996c3305ddbb31093d2411fd360acd20b0f22d5db16f124f37"}
Jul 06 15:43:54.936709 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.936595    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:54.938145 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.938118    2567 generic.go:356] "Generic (PLEG): container finished" podID="8d6a28b8-d5d2-486a-9259-986e1854dccf" containerID="f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c" exitCode=0
Jul 06 15:43:54.938263 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.938158    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:54.939141 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.939115    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:54.940200 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.940174    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn" event={"ID":"7d67df61-0e35-43f7-9edc-93bcbc9c49f5","Type":"ContainerStarted","Data":"bead0550addb304f5d40929bd74ee15a5be48541e39d4801d8495501a4027ab7"}
Jul 06 15:43:54.940324 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.940224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn" event={"ID":"7d67df61-0e35-43f7-9edc-93bcbc9c49f5","Type":"ContainerStarted","Data":"83cad9a8c33d5eda32b507ad4e8bad53d2de0ef89f9b731cd3b69ae1fc8bd5a6"}
Jul 06 15:43:54.940324 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.940255    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:43:54.941623 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.941590    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.941939 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.941913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc" event={"ID":"6587014d-7185-484b-a6e8-15a29fb2b75f","Type":"ContainerStarted","Data":"48cfa474e2c50c38c650d42fcee1d954e8ecfbb083961c76a3142db87c10a7d4"}
Jul 06 15:43:54.944085 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.944046    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:54.946899 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.946416    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:54.950084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.948908    2567 scope.go:117] "RemoveContainer" containerID="9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730"
Jul 06 15:43:54.950084 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.949894    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730\": container with ID starting with 9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730 not found: ID does not exist" containerID="9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730"
Jul 06 15:43:54.950084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.949937    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730"} err="failed to get container status \"9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730\": rpc error: code = NotFound desc = could not find container \"9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730\": container with ID starting with 9720f0121a693b2e7971d8cd62add7464711660783f3265fbcbf7aae5b0d6730 not found: ID does not exist"
Jul 06 15:43:54.950084 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.949961    2567 scope.go:117] "RemoveContainer" containerID="f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c"
Jul 06 15:43:54.954244 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.954218    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6a28b8_d5d2_486a_9259_986e1854dccf.slice/crio-1c695d2751fb51978aadd04deba22aa5af04a1112eda709a22be0fc801efc4c7\": RecentStats: unable to find data in memory cache]"
Jul 06 15:43:54.958949 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.958934    2567 scope.go:117] "RemoveContainer" containerID="f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c"
Jul 06 15:43:54.959197 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:54.959175    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c\": container with ID starting with f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c not found: ID does not exist" containerID="f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c"
Jul 06 15:43:54.959288 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.959202    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c"} err="failed to get container status \"f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c\": rpc error: code = NotFound desc = could not find container \"f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c\": container with ID starting with f13f125b9413f91e5d4ba38f23a0fa4f988587303608772693fa26bd84a79e0c not found: ID does not exist"
Jul 06 15:43:54.975363 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:54.975313    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn" podStartSLOduration=13.975301667 podStartE2EDuration="13.975301667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:54.9740987 +0000 UTC m=+601.844145871" watchObservedRunningTime="2026-07-06 15:43:54.975301667 +0000 UTC m=+601.845348834"
Jul 06 15:43:55.014100 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:55.014055    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" podStartSLOduration=14.014042266 podStartE2EDuration="14.014042266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:55.013023829 +0000 UTC m=+601.883070997" watchObservedRunningTime="2026-07-06 15:43:55.014042266 +0000 UTC m=+601.884089434"
Jul 06 15:43:55.019864 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:55.019838    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g8bcg\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g8bcg"
Jul 06 15:43:55.048013 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:55.047961    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-llrkc" podStartSLOduration=2.816827479 podStartE2EDuration="25.047945304s" totalImagesPullingTime="22.231117825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:55.047409905 +0000 UTC m=+601.917457071" watchObservedRunningTime="2026-07-06 15:43:55.047945304 +0000 UTC m=+601.917992472"
Jul 06 15:43:55.049970 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:55.049946    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xxbm6\" is forbidden: User \"system:node:ip-10-0-138-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-21.ec2.internal' and this object" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xxbm6"
Jul 06 15:43:55.664317 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:55.664284    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6a28b8-d5d2-486a-9259-986e1854dccf" path="/var/lib/kubelet/pods/8d6a28b8-d5d2-486a-9259-986e1854dccf/volumes"
Jul 06 15:43:55.664712 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:55.664620    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e699ab7-8bf1-4fba-b842-5bc488af6df5" path="/var/lib/kubelet/pods/9e699ab7-8bf1-4fba-b842-5bc488af6df5/volumes"
Jul 06 15:43:59.166066 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.166025    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:43:59.166634 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.166215    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" podUID="1b302d83-cd01-4fde-8218-fe115256ea00" containerName="manager" containerID="cri-o://740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc" gracePeriod=10
Jul 06 15:43:59.168243 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.168221    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:59.458090 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.458064    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:59.629240 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.629210    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf6rg\" (UniqueName: \"kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg\") pod \"1b302d83-cd01-4fde-8218-fe115256ea00\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") "
Jul 06 15:43:59.629417 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.629268    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume\") pod \"1b302d83-cd01-4fde-8218-fe115256ea00\" (UID: \"1b302d83-cd01-4fde-8218-fe115256ea00\") "
Jul 06 15:43:59.629693 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.629647    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b302d83-cd01-4fde-8218-fe115256ea00" (UID: "1b302d83-cd01-4fde-8218-fe115256ea00"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:43:59.631509 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.631490    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg" (OuterVolumeSpecName: "kube-api-access-mf6rg") pod "1b302d83-cd01-4fde-8218-fe115256ea00" (UID: "1b302d83-cd01-4fde-8218-fe115256ea00"). InnerVolumeSpecName "kube-api-access-mf6rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:43:59.730558 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.730526    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf6rg\" (UniqueName: \"kubernetes.io/projected/1b302d83-cd01-4fde-8218-fe115256ea00-kube-api-access-mf6rg\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:43:59.730558 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.730553    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b302d83-cd01-4fde-8218-fe115256ea00-extensions-socket-volume\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:43:59.966758 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.966718    2567 generic.go:356] "Generic (PLEG): container finished" podID="1b302d83-cd01-4fde-8218-fe115256ea00" containerID="740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc" exitCode=0
Jul 06 15:43:59.966933 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.966778    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"
Jul 06 15:43:59.966933 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.966796    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" event={"ID":"1b302d83-cd01-4fde-8218-fe115256ea00","Type":"ContainerDied","Data":"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"}
Jul 06 15:43:59.966933 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.966858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c" event={"ID":"1b302d83-cd01-4fde-8218-fe115256ea00","Type":"ContainerDied","Data":"35cbbeae62a232996c3305ddbb31093d2411fd360acd20b0f22d5db16f124f37"}
Jul 06 15:43:59.966933 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.966882    2567 scope.go:117] "RemoveContainer" containerID="740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"
Jul 06 15:43:59.975086 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.975068    2567 scope.go:117] "RemoveContainer" containerID="740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"
Jul 06 15:43:59.975354 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:43:59.975336    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc\": container with ID starting with 740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc not found: ID does not exist" containerID="740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"
Jul 06 15:43:59.975406 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.975362    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc"} err="failed to get container status \"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc\": rpc error: code = NotFound desc = could not find container \"740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc\": container with ID starting with 740817c2d482566eff125c2f246487cbbd163723d084c20547fe901777360ecc not found: ID does not exist"
Jul 06 15:43:59.994024 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.993960    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:43:59.998906 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:43:59.998883    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6dp6c"]
Jul 06 15:44:01.663300 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:01.663271    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b302d83-cd01-4fde-8218-fe115256ea00" path="/var/lib/kubelet/pods/1b302d83-cd01-4fde-8218-fe115256ea00/volumes"
Jul 06 15:44:05.950812 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:05.950775    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5wnfn"
Jul 06 15:44:15.697832 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.697754    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"]
Jul 06 15:44:15.698275 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.698143    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b302d83-cd01-4fde-8218-fe115256ea00" containerName="manager"
Jul 06 15:44:15.698275 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.698159    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b302d83-cd01-4fde-8218-fe115256ea00" containerName="manager"
Jul 06 15:44:15.698275 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.698236    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b302d83-cd01-4fde-8218-fe115256ea00" containerName="manager"
Jul 06 15:44:15.722009 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.721975    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.723878 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.723851    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"]
Jul 06 15:44:15.724496 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.724474    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lx88b\""
Jul 06 15:44:15.750696 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750735    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqj6\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-kube-api-access-9gqj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.750919 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.750908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851258 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851258 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqj6\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-kube-api-access-9gqj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851394    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851475 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851802 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851487    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851802 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851733    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851802 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851785    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851948 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851896    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.851988 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.851951    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.852282 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.852264    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.853973 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.853946    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.854487 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.854462    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.868857 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.868832    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqj6\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-kube-api-access-9gqj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:15.874968 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:15.874943    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5c6vz\" (UID: \"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:16.033741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:16.033623    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:16.254486 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:16.254459    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"]
Jul 06 15:44:16.256234 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:44:16.256209    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea629bd_0b5c_44bf_bdfb_5ad5d3cbe3a0.slice/crio-7fc5bd6fa79062e0e0ecbe7b2e0f4ce38d8c844631e0dc8a225856a355562c23 WatchSource:0}: Error finding container 7fc5bd6fa79062e0e0ecbe7b2e0f4ce38d8c844631e0dc8a225856a355562c23: Status 404 returned error can't find the container with id 7fc5bd6fa79062e0e0ecbe7b2e0f4ce38d8c844631e0dc8a225856a355562c23
Jul 06 15:44:16.258561 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:16.258527    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:44:16.258635 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:16.258598    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:44:16.258635 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:16.258627    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:44:17.042811 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:17.042772    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" event={"ID":"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0","Type":"ContainerStarted","Data":"e13dce2e8af4b7f969d0f5af29aa4648444be0a737487a7f083e539000edb610"}
Jul 06 15:44:17.043235 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:17.042836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" event={"ID":"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0","Type":"ContainerStarted","Data":"7fc5bd6fa79062e0e0ecbe7b2e0f4ce38d8c844631e0dc8a225856a355562c23"}
Jul 06 15:44:17.083586 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:17.083540    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" podStartSLOduration=2.083526174 podStartE2EDuration="2.083526174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:17.080823292 +0000 UTC m=+623.950870461" watchObservedRunningTime="2026-07-06 15:44:17.083526174 +0000 UTC m=+623.953573403"
Jul 06 15:44:18.034393 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.034356    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:18.039600 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.039575    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:18.047914 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.047892    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:18.048789 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.048768    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:44:18.238935 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.238904    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:18.267832 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.267806    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:18.267993 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.267894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.274411 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.274389    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 15:44:18.343569 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.343492    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:18.368508 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.368478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzzx\" (UniqueName: \"kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.368651 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.368519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.469529 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.469490    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzzx\" (UniqueName: \"kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.469737 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.469536    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.470179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.470156    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.480440 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.480420    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzzx\" (UniqueName: \"kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx\") pod \"limitador-limitador-64c94fd64f-7gxcf\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.579496 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.579456    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:18.749500 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:18.749428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:18.751806 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:44:18.751778    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e7bb96_06c9_49fd_8978_c7021ee5276c.slice/crio-9121c01e9369569a70ee2d4af14fdc74c30c52a74df63ebef9a06001d7786e5a WatchSource:0}: Error finding container 9121c01e9369569a70ee2d4af14fdc74c30c52a74df63ebef9a06001d7786e5a: Status 404 returned error can't find the container with id 9121c01e9369569a70ee2d4af14fdc74c30c52a74df63ebef9a06001d7786e5a
Jul 06 15:44:19.054761 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.054728    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" event={"ID":"e1e7bb96-06c9-49fd-8978-c7021ee5276c","Type":"ContainerStarted","Data":"9121c01e9369569a70ee2d4af14fdc74c30c52a74df63ebef9a06001d7786e5a"}
Jul 06 15:44:19.747401 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.747212    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:19.751878 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.751851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:19.756625 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.756311    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5g5h8\""
Jul 06 15:44:19.764974 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.764924    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:19.780950 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.780911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtht5\" (UniqueName: \"kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5\") pod \"authorino-75d556bd77-grg5h\" (UID: \"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab\") " pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:19.881834 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.881795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtht5\" (UniqueName: \"kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5\") pod \"authorino-75d556bd77-grg5h\" (UID: \"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab\") " pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:19.891427 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:19.891361    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtht5\" (UniqueName: \"kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5\") pod \"authorino-75d556bd77-grg5h\" (UID: \"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab\") " pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:20.068425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:20.068352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:20.231386 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:20.231349    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:21.068139 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:21.068092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grg5h" event={"ID":"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab","Type":"ContainerStarted","Data":"479b0b44aba792e1d6605ca9c63b08d0f7ae1b10548f1973ba2042aa46c9dcd0"}
Jul 06 15:44:23.082264 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:23.082154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" event={"ID":"e1e7bb96-06c9-49fd-8978-c7021ee5276c","Type":"ContainerStarted","Data":"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"}
Jul 06 15:44:23.082621 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:23.082442    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:23.101482 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:23.101396    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" podStartSLOduration=1.206577623 podStartE2EDuration="5.101375951s" totalImagesPullingTime="3.894798328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:23.100948479 +0000 UTC m=+629.970995655" watchObservedRunningTime="2026-07-06 15:44:23.101375951 +0000 UTC m=+629.971423120"
Jul 06 15:44:25.093970 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:25.093931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grg5h" event={"ID":"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab","Type":"ContainerStarted","Data":"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"}
Jul 06 15:44:25.123447 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:25.123396    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-grg5h" podStartSLOduration=1.747792134 podStartE2EDuration="6.123379639s" totalImagesPullingTime="4.375587505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:25.121592469 +0000 UTC m=+631.991639629" watchObservedRunningTime="2026-07-06 15:44:25.123379639 +0000 UTC m=+631.993426807"
Jul 06 15:44:33.238306 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.238263    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:33.238805 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.238497    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" podUID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" containerName="limitador" containerID="cri-o://2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3" gracePeriod=30
Jul 06 15:44:33.239168 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.239149    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:33.796736 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.796679    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:33.905485 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.905450    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file\") pod \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") "
Jul 06 15:44:33.905702 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.905528    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfzzx\" (UniqueName: \"kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx\") pod \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\" (UID: \"e1e7bb96-06c9-49fd-8978-c7021ee5276c\") "
Jul 06 15:44:33.905902 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.905876    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file" (OuterVolumeSpecName: "config-file") pod "e1e7bb96-06c9-49fd-8978-c7021ee5276c" (UID: "e1e7bb96-06c9-49fd-8978-c7021ee5276c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:44:33.907765 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:33.907745    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx" (OuterVolumeSpecName: "kube-api-access-pfzzx") pod "e1e7bb96-06c9-49fd-8978-c7021ee5276c" (UID: "e1e7bb96-06c9-49fd-8978-c7021ee5276c"). InnerVolumeSpecName "kube-api-access-pfzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:44:34.006312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.006276    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfzzx\" (UniqueName: \"kubernetes.io/projected/e1e7bb96-06c9-49fd-8978-c7021ee5276c-kube-api-access-pfzzx\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:44:34.006312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.006305    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1e7bb96-06c9-49fd-8978-c7021ee5276c-config-file\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:44:34.134236 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.134141    2567 generic.go:356] "Generic (PLEG): container finished" podID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" containerID="2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3" exitCode=0
Jul 06 15:44:34.134236 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.134190    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"
Jul 06 15:44:34.134443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.134194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" event={"ID":"e1e7bb96-06c9-49fd-8978-c7021ee5276c","Type":"ContainerDied","Data":"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"}
Jul 06 15:44:34.134443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.134325    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7gxcf" event={"ID":"e1e7bb96-06c9-49fd-8978-c7021ee5276c","Type":"ContainerDied","Data":"9121c01e9369569a70ee2d4af14fdc74c30c52a74df63ebef9a06001d7786e5a"}
Jul 06 15:44:34.134443 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.134348    2567 scope.go:117] "RemoveContainer" containerID="2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"
Jul 06 15:44:34.143384 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.143363    2567 scope.go:117] "RemoveContainer" containerID="2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"
Jul 06 15:44:34.143657 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:44:34.143638    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3\": container with ID starting with 2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3 not found: ID does not exist" containerID="2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"
Jul 06 15:44:34.143746 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.143683    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3"} err="failed to get container status \"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3\": rpc error: code = NotFound desc = could not find container \"2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3\": container with ID starting with 2ba9b34b9c451bae47a5c776c7300ff0243c664d12113a7cc207ec38a4197fd3 not found: ID does not exist"
Jul 06 15:44:34.155586 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.155563    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:34.159636 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:34.159617    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7gxcf"]
Jul 06 15:44:35.664458 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:35.664420    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" path="/var/lib/kubelet/pods/e1e7bb96-06c9-49fd-8978-c7021ee5276c/volumes"
Jul 06 15:44:39.831615 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.831587    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-twzdq"]
Jul 06 15:44:39.832110 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.832050    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" containerName="limitador"
Jul 06 15:44:39.832110 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.832064    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" containerName="limitador"
Jul 06 15:44:39.832217 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.832118    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e7bb96-06c9-49fd-8978-c7021ee5276c" containerName="limitador"
Jul 06 15:44:39.835179 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.835136    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:39.837650 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.837626    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 15:44:39.837806 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.837653    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cfnb6\""
Jul 06 15:44:39.844823 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.844801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-twzdq"]
Jul 06 15:44:39.958284 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.958251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k558\" (UniqueName: \"kubernetes.io/projected/93467ae9-a059-41cc-a755-10d176a0d407-kube-api-access-8k558\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:39.958460 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:39.958302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93467ae9-a059-41cc-a755-10d176a0d407-data\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.059640 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.059600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k558\" (UniqueName: \"kubernetes.io/projected/93467ae9-a059-41cc-a755-10d176a0d407-kube-api-access-8k558\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.059640 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.059648    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93467ae9-a059-41cc-a755-10d176a0d407-data\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.060001 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.059983    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93467ae9-a059-41cc-a755-10d176a0d407-data\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.068958 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.068935    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k558\" (UniqueName: \"kubernetes.io/projected/93467ae9-a059-41cc-a755-10d176a0d407-kube-api-access-8k558\") pod \"postgres-69fb4594cf-twzdq\" (UID: \"93467ae9-a059-41cc-a755-10d176a0d407\") " pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.146036 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.145945    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:40.286299 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:40.286271    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-twzdq"]
Jul 06 15:44:40.288940 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:44:40.288909    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93467ae9_a059_41cc_a755_10d176a0d407.slice/crio-64b8b2af7e1c95da60dcbb11a59331cd59adc32e5a76c79f861367b04e902164 WatchSource:0}: Error finding container 64b8b2af7e1c95da60dcbb11a59331cd59adc32e5a76c79f861367b04e902164: Status 404 returned error can't find the container with id 64b8b2af7e1c95da60dcbb11a59331cd59adc32e5a76c79f861367b04e902164
Jul 06 15:44:41.168803 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:41.168768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-twzdq" event={"ID":"93467ae9-a059-41cc-a755-10d176a0d407","Type":"ContainerStarted","Data":"64b8b2af7e1c95da60dcbb11a59331cd59adc32e5a76c79f861367b04e902164"}
Jul 06 15:44:46.198120 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:46.198081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-twzdq" event={"ID":"93467ae9-a059-41cc-a755-10d176a0d407","Type":"ContainerStarted","Data":"023afa3e730fec332a10b7fa253507f852331b397dba829130d2c1386209f0cd"}
Jul 06 15:44:46.198576 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:46.198185    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:46.215094 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:46.215048    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-twzdq" podStartSLOduration=1.7752310059999998 podStartE2EDuration="7.215033867s" totalImagesPullingTime="5.439802861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:46.214054956 +0000 UTC m=+653.084102127" watchObservedRunningTime="2026-07-06 15:44:46.215033867 +0000 UTC m=+653.085081034"
Jul 06 15:44:52.233344 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:52.233313    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-twzdq"
Jul 06 15:44:54.271508 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.271468    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:54.271959 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.271692    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-grg5h" podUID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" containerName="authorino" containerID="cri-o://d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac" gracePeriod=30
Jul 06 15:44:54.527117 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.527054    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:54.582141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.582110    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtht5\" (UniqueName: \"kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5\") pod \"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab\" (UID: \"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab\") "
Jul 06 15:44:54.584295 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.584269    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5" (OuterVolumeSpecName: "kube-api-access-jtht5") pod "8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" (UID: "8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab"). InnerVolumeSpecName "kube-api-access-jtht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:44:54.682971 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:54.682937    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtht5\" (UniqueName: \"kubernetes.io/projected/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab-kube-api-access-jtht5\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:44:55.239755 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.239718    2567 generic.go:356] "Generic (PLEG): container finished" podID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" containerID="d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac" exitCode=0
Jul 06 15:44:55.239952 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.239760    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-grg5h"
Jul 06 15:44:55.239952 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.239812    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grg5h" event={"ID":"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab","Type":"ContainerDied","Data":"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"}
Jul 06 15:44:55.239952 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.239858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-grg5h" event={"ID":"8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab","Type":"ContainerDied","Data":"479b0b44aba792e1d6605ca9c63b08d0f7ae1b10548f1973ba2042aa46c9dcd0"}
Jul 06 15:44:55.239952 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.239880    2567 scope.go:117] "RemoveContainer" containerID="d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"
Jul 06 15:44:55.249081 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.249064    2567 scope.go:117] "RemoveContainer" containerID="d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"
Jul 06 15:44:55.249322 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:44:55.249305    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac\": container with ID starting with d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac not found: ID does not exist" containerID="d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"
Jul 06 15:44:55.249382 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.249330    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac"} err="failed to get container status \"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac\": rpc error: code = NotFound desc = could not find container \"d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac\": container with ID starting with d220c2b9839df00840b1e1338d83f2d10e974e801df9ae182422ecb6300d94ac not found: ID does not exist"
Jul 06 15:44:55.273489 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.273455    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:55.277512 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.277490    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-grg5h"]
Jul 06 15:44:55.665076 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:55.664993    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" path="/var/lib/kubelet/pods/8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab/volumes"
Jul 06 15:44:59.072700 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.072651    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:44:59.073074 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.073007    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" containerName="authorino"
Jul 06 15:44:59.073074 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.073019    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" containerName="authorino"
Jul 06 15:44:59.073144 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.073082    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c15fcc9-da95-44f2-ba0b-8aeb1a6d87ab" containerName="authorino"
Jul 06 15:44:59.276042 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.276013    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:44:59.276042 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.276052    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:44:59.276260 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.276143    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.279953 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.279933    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 15:44:59.280798 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.280780    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jwvr8\""
Jul 06 15:44:59.280850 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.280819    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 15:44:59.301247 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.301214    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:44:59.301400 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.301329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.421917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.421835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcn2g\" (UniqueName: \"kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.421917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.421870    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z695\" (UniqueName: \"kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.421917 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.421913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.422136 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.421952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.523031 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.522989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcn2g\" (UniqueName: \"kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.523031 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.523035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z695\" (UniqueName: \"kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.523293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.523082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.523293 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.523103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.525880 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.525853    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.525982 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.525955    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.533397 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.533368    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcn2g\" (UniqueName: \"kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g\") pod \"maas-controller-94f47f95b-5hj8l\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") " pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.533512 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.533367    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z695\" (UniqueName: \"kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695\") pod \"maas-controller-cd68b496-cqscj\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") " pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.586116 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.586085    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:44:59.611736 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.611712    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:44:59.760103 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.760069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:44:59.763285 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:44:59.763258    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d418cda_718a_477a_ba5d_ffef57f38f87.slice/crio-304212dc26d966f46086633fdd4d1cc0378bd7ff8f3646bcf4d1aa0822bdd221 WatchSource:0}: Error finding container 304212dc26d966f46086633fdd4d1cc0378bd7ff8f3646bcf4d1aa0822bdd221: Status 404 returned error can't find the container with id 304212dc26d966f46086633fdd4d1cc0378bd7ff8f3646bcf4d1aa0822bdd221
Jul 06 15:44:59.789721 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:44:59.789696    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:44:59.790846 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:44:59.790824    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e1d8e4_6e53_49ae_98a9_3c563b58c520.slice/crio-056489c2ad54c3d2ea95a1d20921e0e682af2465f85c583cb482ea31bc535de1 WatchSource:0}: Error finding container 056489c2ad54c3d2ea95a1d20921e0e682af2465f85c583cb482ea31bc535de1: Status 404 returned error can't find the container with id 056489c2ad54c3d2ea95a1d20921e0e682af2465f85c583cb482ea31bc535de1
Jul 06 15:45:00.263625 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:00.263587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-5hj8l" event={"ID":"64e1d8e4-6e53-49ae-98a9-3c563b58c520","Type":"ContainerStarted","Data":"056489c2ad54c3d2ea95a1d20921e0e682af2465f85c583cb482ea31bc535de1"}
Jul 06 15:45:00.264817 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:00.264793    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cd68b496-cqscj" event={"ID":"3d418cda-718a-477a-ba5d-ffef57f38f87","Type":"ContainerStarted","Data":"304212dc26d966f46086633fdd4d1cc0378bd7ff8f3646bcf4d1aa0822bdd221"}
Jul 06 15:45:02.277868 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.277828    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-5hj8l" event={"ID":"64e1d8e4-6e53-49ae-98a9-3c563b58c520","Type":"ContainerStarted","Data":"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"}
Jul 06 15:45:02.278322 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.277969    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:45:02.279546 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.279519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cd68b496-cqscj" event={"ID":"3d418cda-718a-477a-ba5d-ffef57f38f87","Type":"ContainerStarted","Data":"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"}
Jul 06 15:45:02.279710 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.279657    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:45:02.316188 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.316141    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-94f47f95b-5hj8l" podStartSLOduration=1.28636182 podStartE2EDuration="3.316124056s" totalImagesPullingTime="2.029762236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:02.316094657 +0000 UTC m=+669.186141824" watchObservedRunningTime="2026-07-06 15:45:02.316124056 +0000 UTC m=+669.186171223"
Jul 06 15:45:02.342216 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:02.342164    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cd68b496-cqscj" podStartSLOduration=1.288988647 podStartE2EDuration="3.3421484s" totalImagesPullingTime="2.053159753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:02.340893894 +0000 UTC m=+669.210941063" watchObservedRunningTime="2026-07-06 15:45:02.3421484 +0000 UTC m=+669.212195559"
Jul 06 15:45:13.290048 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.290017    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:45:13.290648 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.290629    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:45:13.376614 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.376577    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:45:13.376841 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.376815    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-cd68b496-cqscj" podUID="3d418cda-718a-477a-ba5d-ffef57f38f87" containerName="manager" containerID="cri-o://f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502" gracePeriod=30
Jul 06 15:45:13.622552 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.622528    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:45:13.751299 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.751267    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert\") pod \"3d418cda-718a-477a-ba5d-ffef57f38f87\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") "
Jul 06 15:45:13.751299 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.751302    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z695\" (UniqueName: \"kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695\") pod \"3d418cda-718a-477a-ba5d-ffef57f38f87\" (UID: \"3d418cda-718a-477a-ba5d-ffef57f38f87\") "
Jul 06 15:45:13.753594 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.753568    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3d418cda-718a-477a-ba5d-ffef57f38f87" (UID: "3d418cda-718a-477a-ba5d-ffef57f38f87"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:45:13.753697 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.753616    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695" (OuterVolumeSpecName: "kube-api-access-6z695") pod "3d418cda-718a-477a-ba5d-ffef57f38f87" (UID: "3d418cda-718a-477a-ba5d-ffef57f38f87"). InnerVolumeSpecName "kube-api-access-6z695". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:45:13.852798 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.852711    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d418cda-718a-477a-ba5d-ffef57f38f87-webhook-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:45:13.852798 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:13.852739    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z695\" (UniqueName: \"kubernetes.io/projected/3d418cda-718a-477a-ba5d-ffef57f38f87-kube-api-access-6z695\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:45:14.331894 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.331859    2567 generic.go:356] "Generic (PLEG): container finished" podID="3d418cda-718a-477a-ba5d-ffef57f38f87" containerID="f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502" exitCode=0
Jul 06 15:45:14.332349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.331903    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cd68b496-cqscj"
Jul 06 15:45:14.332349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.331923    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cd68b496-cqscj" event={"ID":"3d418cda-718a-477a-ba5d-ffef57f38f87","Type":"ContainerDied","Data":"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"}
Jul 06 15:45:14.332349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.331976    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cd68b496-cqscj" event={"ID":"3d418cda-718a-477a-ba5d-ffef57f38f87","Type":"ContainerDied","Data":"304212dc26d966f46086633fdd4d1cc0378bd7ff8f3646bcf4d1aa0822bdd221"}
Jul 06 15:45:14.332349 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.332007    2567 scope.go:117] "RemoveContainer" containerID="f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"
Jul 06 15:45:14.340406 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.340389    2567 scope.go:117] "RemoveContainer" containerID="f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"
Jul 06 15:45:14.340652 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:45:14.340635    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502\": container with ID starting with f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502 not found: ID does not exist" containerID="f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"
Jul 06 15:45:14.340719 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.340676    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502"} err="failed to get container status \"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502\": rpc error: code = NotFound desc = could not find container \"f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502\": container with ID starting with f2c7368f3840ddb86012a4f5e9470c58621f926be452859af0eec0185ccfd502 not found: ID does not exist"
Jul 06 15:45:14.354062 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.354041    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:45:14.359194 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:14.359171    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cd68b496-cqscj"]
Jul 06 15:45:15.664412 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:15.664378    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d418cda-718a-477a-ba5d-ffef57f38f87" path="/var/lib/kubelet/pods/3d418cda-718a-477a-ba5d-ffef57f38f87/volumes"
Jul 06 15:45:34.086699 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.086649    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5d86f67b86-82gjj"]
Jul 06 15:45:34.087083 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.087000    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d418cda-718a-477a-ba5d-ffef57f38f87" containerName="manager"
Jul 06 15:45:34.087083 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.087012    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d418cda-718a-477a-ba5d-ffef57f38f87" containerName="manager"
Jul 06 15:45:34.087083 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.087070    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d418cda-718a-477a-ba5d-ffef57f38f87" containerName="manager"
Jul 06 15:45:34.090154 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.090138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.092488 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.092465    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 06 15:45:34.092686 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.092646    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hz5zw\""
Jul 06 15:45:34.102454 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.102430    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5d86f67b86-82gjj"]
Jul 06 15:45:34.120641 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.120609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1eab4a6-e317-4f6c-8541-c1114a01b86d-openshift-service-ca\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.120827 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.120752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1eab4a6-e317-4f6c-8541-c1114a01b86d-maas-api-tls\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.120827 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.120796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpt6\" (UniqueName: \"kubernetes.io/projected/d1eab4a6-e317-4f6c-8541-c1114a01b86d-kube-api-access-7wpt6\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.222099 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.222062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1eab4a6-e317-4f6c-8541-c1114a01b86d-openshift-service-ca\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.222270 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.222105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1eab4a6-e317-4f6c-8541-c1114a01b86d-maas-api-tls\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.222270 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.222132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpt6\" (UniqueName: \"kubernetes.io/projected/d1eab4a6-e317-4f6c-8541-c1114a01b86d-kube-api-access-7wpt6\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.222704 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.222657    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1eab4a6-e317-4f6c-8541-c1114a01b86d-openshift-service-ca\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.224622 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.224598    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1eab4a6-e317-4f6c-8541-c1114a01b86d-maas-api-tls\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.231338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.231310    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpt6\" (UniqueName: \"kubernetes.io/projected/d1eab4a6-e317-4f6c-8541-c1114a01b86d-kube-api-access-7wpt6\") pod \"maas-api-5d86f67b86-82gjj\" (UID: \"d1eab4a6-e317-4f6c-8541-c1114a01b86d\") " pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.400134 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.400044    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:34.546888 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.546829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5d86f67b86-82gjj"]
Jul 06 15:45:34.548569 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:45:34.548541    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1eab4a6_e317_4f6c_8541_c1114a01b86d.slice/crio-3c0c986c12c28251b661c86154c8f2854680aeeab36ade902a9a84e0a2438642 WatchSource:0}: Error finding container 3c0c986c12c28251b661c86154c8f2854680aeeab36ade902a9a84e0a2438642: Status 404 returned error can't find the container with id 3c0c986c12c28251b661c86154c8f2854680aeeab36ade902a9a84e0a2438642
Jul 06 15:45:34.657537 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.655637    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpbqn"]
Jul 06 15:45:34.664003 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.662159    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.665245 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.665221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-82mdt\""
Jul 06 15:45:34.665245 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.665230    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 15:45:34.671545 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.671520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpbqn"]
Jul 06 15:45:34.726786 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.726756    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4bk9\" (UniqueName: \"kubernetes.io/projected/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-kube-api-access-f4bk9\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.726977 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.726955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.742550 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.742523    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-f8pfl"]
Jul 06 15:45:34.746107 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.746092    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.756622 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.756594    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-f8pfl"]
Jul 06 15:45:34.828121 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.828087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.828286 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.828142    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6766022a-d3f6-46af-b15d-9c854546db52-plugins-config-volume\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.828286 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.828238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4bk9\" (UniqueName: \"kubernetes.io/projected/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-kube-api-access-f4bk9\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.828368 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.828306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qp6\" (UniqueName: \"kubernetes.io/projected/6766022a-d3f6-46af-b15d-9c854546db52-kube-api-access-d4qp6\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.828904 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.828887    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.837142 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.837114    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4bk9\" (UniqueName: \"kubernetes.io/projected/d31f62bf-ab10-497c-8e2b-d3c2bdc99387-kube-api-access-f4bk9\") pod \"payload-pre-processing-7964457696-kpbqn\" (UID: \"d31f62bf-ab10-497c-8e2b-d3c2bdc99387\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:34.929419 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.929331    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6766022a-d3f6-46af-b15d-9c854546db52-plugins-config-volume\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.929419 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.929407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qp6\" (UniqueName: \"kubernetes.io/projected/6766022a-d3f6-46af-b15d-9c854546db52-kube-api-access-d4qp6\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.930118 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.930099    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6766022a-d3f6-46af-b15d-9c854546db52-plugins-config-volume\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.937624 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.937596    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qp6\" (UniqueName: \"kubernetes.io/projected/6766022a-d3f6-46af-b15d-9c854546db52-kube-api-access-d4qp6\") pod \"payload-processing-58f86c7798-f8pfl\" (UID: \"6766022a-d3f6-46af-b15d-9c854546db52\") " pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:34.975809 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:34.975778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:35.059623 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.059590    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:35.116825 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.116801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpbqn"]
Jul 06 15:45:35.119505 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:45:35.119473    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31f62bf_ab10_497c_8e2b_d3c2bdc99387.slice/crio-066f9ae22023f021e25840b576b771ba4e56d488f5b78ee9993249eca23f15fb WatchSource:0}: Error finding container 066f9ae22023f021e25840b576b771ba4e56d488f5b78ee9993249eca23f15fb: Status 404 returned error can't find the container with id 066f9ae22023f021e25840b576b771ba4e56d488f5b78ee9993249eca23f15fb
Jul 06 15:45:35.211839 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.211808    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-f8pfl"]
Jul 06 15:45:35.214584 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:45:35.214547    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6766022a_d3f6_46af_b15d_9c854546db52.slice/crio-9b7d2a0ad5c0af589f39bee5929659e0a0cb57954990545ccc49d8563e2d6cea WatchSource:0}: Error finding container 9b7d2a0ad5c0af589f39bee5929659e0a0cb57954990545ccc49d8563e2d6cea: Status 404 returned error can't find the container with id 9b7d2a0ad5c0af589f39bee5929659e0a0cb57954990545ccc49d8563e2d6cea
Jul 06 15:45:35.430593 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.430522    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl" event={"ID":"6766022a-d3f6-46af-b15d-9c854546db52","Type":"ContainerStarted","Data":"9b7d2a0ad5c0af589f39bee5929659e0a0cb57954990545ccc49d8563e2d6cea"}
Jul 06 15:45:35.431880 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.431851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn" event={"ID":"d31f62bf-ab10-497c-8e2b-d3c2bdc99387","Type":"ContainerStarted","Data":"066f9ae22023f021e25840b576b771ba4e56d488f5b78ee9993249eca23f15fb"}
Jul 06 15:45:35.433093 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:35.433064    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5d86f67b86-82gjj" event={"ID":"d1eab4a6-e317-4f6c-8541-c1114a01b86d","Type":"ContainerStarted","Data":"3c0c986c12c28251b661c86154c8f2854680aeeab36ade902a9a84e0a2438642"}
Jul 06 15:45:36.441178 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:36.441094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5d86f67b86-82gjj" event={"ID":"d1eab4a6-e317-4f6c-8541-c1114a01b86d","Type":"ContainerStarted","Data":"e7386f0c738e209c46f345bb1377d3b1094b72b25d852f868488ea511cbdee26"}
Jul 06 15:45:36.441612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:36.441333    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:36.460523 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:36.460477    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5d86f67b86-82gjj" podStartSLOduration=0.906912941 podStartE2EDuration="2.460462361s" totalImagesPullingTime="1.55354942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:36.458877632 +0000 UTC m=+703.328924800" watchObservedRunningTime="2026-07-06 15:45:36.460462361 +0000 UTC m=+703.330509528"
Jul 06 15:45:37.447014 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.446981    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl" event={"ID":"6766022a-d3f6-46af-b15d-9c854546db52","Type":"ContainerStarted","Data":"74e0ec1fcdcc361912a844a8f81aff60cb2fe43250484bd64906d79538018d7d"}
Jul 06 15:45:37.447424 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.447066    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:37.448508 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.448478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn" event={"ID":"d31f62bf-ab10-497c-8e2b-d3c2bdc99387","Type":"ContainerStarted","Data":"6ecf2fd23a20674feb8c76f87e339b2ffa07f9424084c8e554ae256660942609"}
Jul 06 15:45:37.448630 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.448563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:37.465211 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.465171    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl" podStartSLOduration=1.384435177 podStartE2EDuration="3.465158662s" totalImagesPullingTime="2.080723485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:37.463763651 +0000 UTC m=+704.333810820" watchObservedRunningTime="2026-07-06 15:45:37.465158662 +0000 UTC m=+704.335205830"
Jul 06 15:45:37.479275 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:37.479221    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn" podStartSLOduration=1.342076848 podStartE2EDuration="3.479201901s" totalImagesPullingTime="2.137125053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:37.478364997 +0000 UTC m=+704.348412179" watchObservedRunningTime="2026-07-06 15:45:37.479201901 +0000 UTC m=+704.349249073"
Jul 06 15:45:42.453777 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.453747    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5d86f67b86-82gjj"
Jul 06 15:45:42.472985 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.472952    2567 generic.go:356] "Generic (PLEG): container finished" podID="fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0" containerID="e13dce2e8af4b7f969d0f5af29aa4648444be0a737487a7f083e539000edb610" exitCode=0
Jul 06 15:45:42.473110 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.473027    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" event={"ID":"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0","Type":"ContainerDied","Data":"e13dce2e8af4b7f969d0f5af29aa4648444be0a737487a7f083e539000edb610"}
Jul 06 15:45:42.478231 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.473928    2567 scope.go:117] "RemoveContainer" containerID="e13dce2e8af4b7f969d0f5af29aa4648444be0a737487a7f083e539000edb610"
Jul 06 15:45:42.479234 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.479200    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:45:42.479312 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.479297    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:45:42.479356 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:42.479342    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 15:45:43.483695 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:43.483385    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" event={"ID":"fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0","Type":"ContainerStarted","Data":"2efdb1d0cea38bf18f1dbd7db8d76f41892c4fa44f52732bce4a34bd1260113b"}
Jul 06 15:45:44.034425 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:44.034383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:45:44.035531 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:44.035507    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5c6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 15:45:44.035589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:44.035557    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" podUID="fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 15:45:45.034206 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:45.034171    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5c6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 15:45:45.034583 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:45.034240    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" podUID="fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 15:45:46.034534 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:46.034501    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5c6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 15:45:46.034931 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:46.034559    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" podUID="fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 15:45:47.034507 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:47.034468    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5c6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 15:45:47.034700 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:47.034532    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz" podUID="fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 15:45:48.038008 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:48.037974    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:45:48.038445 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:48.038261    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:45:48.038937 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:48.038922    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5c6vz"
Jul 06 15:45:48.454220 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:48.454193    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-f8pfl"
Jul 06 15:45:48.454390 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:48.454353    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-kpbqn"
Jul 06 15:45:58.246374 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.246337    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"]
Jul 06 15:45:58.255051 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.255024    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.258741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.258275    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cmsdq\""
Jul 06 15:45:58.258741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.258275    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 06 15:45:58.258741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.258349    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 15:45:58.258741 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.258276    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 15:45:58.260649 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.260628    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"]
Jul 06 15:45:58.334183 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.334338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b51d3e-371a-469b-b9b5-f763f6cd5177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.334338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.334338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.334338 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkssv\" (UniqueName: \"kubernetes.io/projected/27b51d3e-371a-469b-b9b5-f763f6cd5177-kube-api-access-tkssv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.334476 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.334377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435295 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435295 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b51d3e-371a-469b-b9b5-f763f6cd5177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435500 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435500 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435500 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkssv\" (UniqueName: \"kubernetes.io/projected/27b51d3e-371a-469b-b9b5-f763f6cd5177-kube-api-access-tkssv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435638 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.435909 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435878    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.436033 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435879    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.436033 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.435938    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.437752 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.437727    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b51d3e-371a-469b-b9b5-f763f6cd5177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.438036 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.438020    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b51d3e-371a-469b-b9b5-f763f6cd5177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.443747 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.443729    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkssv\" (UniqueName: \"kubernetes.io/projected/27b51d3e-371a-469b-b9b5-f763f6cd5177-kube-api-access-tkssv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx\" (UID: \"27b51d3e-371a-469b-b9b5-f763f6cd5177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.566047 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.565959    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:45:58.706323 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.706224    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"]
Jul 06 15:45:58.709197 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:45:58.709162    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b51d3e_371a_469b_b9b5_f763f6cd5177.slice/crio-c10d7ca73cf4a7c43476a290a0dcf15b2e5bc677bbde670f7d153b69fa384ab0 WatchSource:0}: Error finding container c10d7ca73cf4a7c43476a290a0dcf15b2e5bc677bbde670f7d153b69fa384ab0: Status 404 returned error can't find the container with id c10d7ca73cf4a7c43476a290a0dcf15b2e5bc677bbde670f7d153b69fa384ab0
Jul 06 15:45:58.710544 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:58.710525    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:45:59.562889 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:45:59.562847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx" event={"ID":"27b51d3e-371a-469b-b9b5-f763f6cd5177","Type":"ContainerStarted","Data":"c10d7ca73cf4a7c43476a290a0dcf15b2e5bc677bbde670f7d153b69fa384ab0"}
Jul 06 15:46:01.575602 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:01.575557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx" event={"ID":"27b51d3e-371a-469b-b9b5-f763f6cd5177","Type":"ContainerStarted","Data":"5bffeaac7ec168252a6574fde87eca1ce58299f9e9fbc1c1c1f9c1c755ee5de6"}
Jul 06 15:46:08.567073 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:08.567029    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:46:08.567506 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:08.567104    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:46:08.579914 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:08.579880    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:46:08.600326 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:08.600273    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx" podStartSLOduration=8.363543248 podStartE2EDuration="10.600254117s" totalImagesPullingTime="2.236710869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:46:01.599560053 +0000 UTC m=+728.469607221" watchObservedRunningTime="2026-07-06 15:46:08.600254117 +0000 UTC m=+735.470301287"
Jul 06 15:46:08.618595 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:08.618572    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx"
Jul 06 15:46:37.542045 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.542007    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"]
Jul 06 15:46:37.545695 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.545647    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.548306 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.548283    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 06 15:46:37.555615 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.555593    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"]
Jul 06 15:46:37.589980 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.589949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7pb\" (UniqueName: \"kubernetes.io/projected/66d2b339-fd7e-4637-b558-31c84779d276-kube-api-access-qp7pb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.589980 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.589992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.590191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.590022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.590191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.590044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d2b339-fd7e-4637-b558-31c84779d276-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.590191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.590135    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.590191 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.590165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691332 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7pb\" (UniqueName: \"kubernetes.io/projected/66d2b339-fd7e-4637-b558-31c84779d276-kube-api-access-qp7pb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691332 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691331    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d2b339-fd7e-4637-b558-31c84779d276-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691589 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691799 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691778    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.691915 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691888    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.692022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.691893    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.693787 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.693765    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d2b339-fd7e-4637-b558-31c84779d276-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.694069 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.694047    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d2b339-fd7e-4637-b558-31c84779d276-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.708452 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.708424    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7pb\" (UniqueName: \"kubernetes.io/projected/66d2b339-fd7e-4637-b558-31c84779d276-kube-api-access-qp7pb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b\" (UID: \"66d2b339-fd7e-4637-b558-31c84779d276\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.856901 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.856821    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:37.993803 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:37.993731    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"]
Jul 06 15:46:37.996051 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:46:37.996028    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2b339_fd7e_4637_b558_31c84779d276.slice/crio-5fbfd7513fe001e772d39c7ee1624fd21a61aa1792827821b6adbc50cb220c4a WatchSource:0}: Error finding container 5fbfd7513fe001e772d39c7ee1624fd21a61aa1792827821b6adbc50cb220c4a: Status 404 returned error can't find the container with id 5fbfd7513fe001e772d39c7ee1624fd21a61aa1792827821b6adbc50cb220c4a
Jul 06 15:46:38.747820 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:38.747726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b" event={"ID":"66d2b339-fd7e-4637-b558-31c84779d276","Type":"ContainerStarted","Data":"bab932bd82f98e35e0d93542d920e11f4703d9e62892c97d7049730c774e37f8"}
Jul 06 15:46:38.747820 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:38.747788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b" event={"ID":"66d2b339-fd7e-4637-b558-31c84779d276","Type":"ContainerStarted","Data":"5fbfd7513fe001e772d39c7ee1624fd21a61aa1792827821b6adbc50cb220c4a"}
Jul 06 15:46:38.767936 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:38.767884    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b" podStartSLOduration=1.285014457 podStartE2EDuration="1.767867189s" totalImagesPullingTime="482.852732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:46:38.766653557 +0000 UTC m=+765.636700726" watchObservedRunningTime="2026-07-06 15:46:38.767867189 +0000 UTC m=+765.637914356"
Jul 06 15:46:47.857760 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:47.857717    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:47.858256 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:47.857785    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:47.870207 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:47.870176    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:46:48.803504 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:46:48.803477    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b"
Jul 06 15:47:06.219876 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.219837    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"]
Jul 06 15:47:06.222609 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.222588    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.225243 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.225220    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 06 15:47:06.237103 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.237069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"]
Jul 06 15:47:06.357409 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.357612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.357612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltd2\" (UniqueName: \"kubernetes.io/projected/8bf23871-faa3-423f-899e-5d79b0b437b4-kube-api-access-cltd2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.357612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.357612 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.357870 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.357623    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23871-faa3-423f-899e-5d79b0b437b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.458933 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.458886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23871-faa3-423f-899e-5d79b0b437b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459146 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.458938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459146 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459146 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltd2\" (UniqueName: \"kubernetes.io/projected/8bf23871-faa3-423f-899e-5d79b0b437b4-kube-api-access-cltd2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459146 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459146 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459471 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459439    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459471 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459465    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.459590 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.459515    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.461464 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.461438    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23871-faa3-423f-899e-5d79b0b437b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.461843 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.461812    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23871-faa3-423f-899e-5d79b0b437b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.468034 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.468010    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltd2\" (UniqueName: \"kubernetes.io/projected/8bf23871-faa3-423f-899e-5d79b0b437b4-kube-api-access-cltd2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj\" (UID: \"8bf23871-faa3-423f-899e-5d79b0b437b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.532729 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.532605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:06.678805 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.678613    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"]
Jul 06 15:47:06.681816 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:47:06.681778    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bf23871_faa3_423f_899e_5d79b0b437b4.slice/crio-e33651bff77764c156f5b4a723a5685f03319773e8aec390bbfa13f3aabfa0e7 WatchSource:0}: Error finding container e33651bff77764c156f5b4a723a5685f03319773e8aec390bbfa13f3aabfa0e7: Status 404 returned error can't find the container with id e33651bff77764c156f5b4a723a5685f03319773e8aec390bbfa13f3aabfa0e7
Jul 06 15:47:06.877756 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:06.877654    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj" event={"ID":"8bf23871-faa3-423f-899e-5d79b0b437b4","Type":"ContainerStarted","Data":"e33651bff77764c156f5b4a723a5685f03319773e8aec390bbfa13f3aabfa0e7"}
Jul 06 15:47:07.883944 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:07.883903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj" event={"ID":"8bf23871-faa3-423f-899e-5d79b0b437b4","Type":"ContainerStarted","Data":"2e88e43463e6c923775fccd09f5a259cc89ac2c7e3715ad8559b55d8145fc79d"}
Jul 06 15:47:07.936507 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:07.936365    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj" podStartSLOduration=1.729893034 podStartE2EDuration="1.936341855s" totalImagesPullingTime="206.448821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:47:07.935763314 +0000 UTC m=+794.805810481" watchObservedRunningTime="2026-07-06 15:47:07.936341855 +0000 UTC m=+794.806389023"
Jul 06 15:47:16.533121 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:16.533032    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:16.533121 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:16.533092    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:16.545762 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:16.545737    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:47:16.937947 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:47:16.937871    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj"
Jul 06 15:49:26.147253 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.147218    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:49:26.147833 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.147410    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-94f47f95b-5hj8l" podUID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" containerName="manager" containerID="cri-o://68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8" gracePeriod=30
Jul 06 15:49:26.398410 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.398349    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:49:26.514420 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.514391    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcn2g\" (UniqueName: \"kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g\") pod \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") "
Jul 06 15:49:26.514588 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.514446    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert\") pod \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\" (UID: \"64e1d8e4-6e53-49ae-98a9-3c563b58c520\") "
Jul 06 15:49:26.516579 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.516550    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "64e1d8e4-6e53-49ae-98a9-3c563b58c520" (UID: "64e1d8e4-6e53-49ae-98a9-3c563b58c520"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:49:26.516717 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.516643    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g" (OuterVolumeSpecName: "kube-api-access-mcn2g") pod "64e1d8e4-6e53-49ae-98a9-3c563b58c520" (UID: "64e1d8e4-6e53-49ae-98a9-3c563b58c520"). InnerVolumeSpecName "kube-api-access-mcn2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:49:26.529864 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.529834    2567 generic.go:356] "Generic (PLEG): container finished" podID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" containerID="68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8" exitCode=0
Jul 06 15:49:26.530022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.529887    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-5hj8l"
Jul 06 15:49:26.530022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.529913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-5hj8l" event={"ID":"64e1d8e4-6e53-49ae-98a9-3c563b58c520","Type":"ContainerDied","Data":"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"}
Jul 06 15:49:26.530022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.529966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-5hj8l" event={"ID":"64e1d8e4-6e53-49ae-98a9-3c563b58c520","Type":"ContainerDied","Data":"056489c2ad54c3d2ea95a1d20921e0e682af2465f85c583cb482ea31bc535de1"}
Jul 06 15:49:26.530022 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.529989    2567 scope.go:117] "RemoveContainer" containerID="68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"
Jul 06 15:49:26.540144 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.540127    2567 scope.go:117] "RemoveContainer" containerID="68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"
Jul 06 15:49:26.540475 ip-10-0-138-21 kubenswrapper[2567]: E0706 15:49:26.540451    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8\": container with ID starting with 68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8 not found: ID does not exist" containerID="68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"
Jul 06 15:49:26.540536 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.540488    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8"} err="failed to get container status \"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8\": rpc error: code = NotFound desc = could not find container \"68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8\": container with ID starting with 68de9a3108e217e08ae669aab0457e26da4f55bb19c16edd12c720b06324a4d8 not found: ID does not exist"
Jul 06 15:49:26.554132 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.554105    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:49:26.561366 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.561340    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-94f47f95b-5hj8l"]
Jul 06 15:49:26.616009 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.615970    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e1d8e4-6e53-49ae-98a9-3c563b58c520-webhook-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:49:26.616009 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:26.616001    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcn2g\" (UniqueName: \"kubernetes.io/projected/64e1d8e4-6e53-49ae-98a9-3c563b58c520-kube-api-access-mcn2g\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 15:49:27.338274 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.338235    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94f47f95b-4wh4b"]
Jul 06 15:49:27.338693 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.338631    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" containerName="manager"
Jul 06 15:49:27.338693 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.338643    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" containerName="manager"
Jul 06 15:49:27.338771 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.338728    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" containerName="manager"
Jul 06 15:49:27.341059 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.341044    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.345825 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.345802    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 15:49:27.345954 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.345859    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jwvr8\""
Jul 06 15:49:27.345954 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.345862    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 15:49:27.355966 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.355941    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94f47f95b-4wh4b"]
Jul 06 15:49:27.425060 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.425015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a142576-4499-48f4-9984-6c1f3b29a62e-webhook-cert\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.425250 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.425139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqh4\" (UniqueName: \"kubernetes.io/projected/3a142576-4499-48f4-9984-6c1f3b29a62e-kube-api-access-tnqh4\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.525725 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.525641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqh4\" (UniqueName: \"kubernetes.io/projected/3a142576-4499-48f4-9984-6c1f3b29a62e-kube-api-access-tnqh4\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.525923 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.525761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a142576-4499-48f4-9984-6c1f3b29a62e-webhook-cert\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.528454 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.528426    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a142576-4499-48f4-9984-6c1f3b29a62e-webhook-cert\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.534926 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.534853    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqh4\" (UniqueName: \"kubernetes.io/projected/3a142576-4499-48f4-9984-6c1f3b29a62e-kube-api-access-tnqh4\") pod \"maas-controller-94f47f95b-4wh4b\" (UID: \"3a142576-4499-48f4-9984-6c1f3b29a62e\") " pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.651277 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.651196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:27.667588 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.667551    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e1d8e4-6e53-49ae-98a9-3c563b58c520" path="/var/lib/kubelet/pods/64e1d8e4-6e53-49ae-98a9-3c563b58c520/volumes"
Jul 06 15:49:27.788154 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:27.788131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94f47f95b-4wh4b"]
Jul 06 15:49:27.789901 ip-10-0-138-21 kubenswrapper[2567]: W0706 15:49:27.789875    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a142576_4499_48f4_9984_6c1f3b29a62e.slice/crio-0124018cefc6c4b7c395b2c0fa0da1cf36c04bd8232e5b34a93c8f730e935bbf WatchSource:0}: Error finding container 0124018cefc6c4b7c395b2c0fa0da1cf36c04bd8232e5b34a93c8f730e935bbf: Status 404 returned error can't find the container with id 0124018cefc6c4b7c395b2c0fa0da1cf36c04bd8232e5b34a93c8f730e935bbf
Jul 06 15:49:28.541141 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:28.541098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-4wh4b" event={"ID":"3a142576-4499-48f4-9984-6c1f3b29a62e","Type":"ContainerStarted","Data":"0124018cefc6c4b7c395b2c0fa0da1cf36c04bd8232e5b34a93c8f730e935bbf"}
Jul 06 15:49:30.552817 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:30.552780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94f47f95b-4wh4b" event={"ID":"3a142576-4499-48f4-9984-6c1f3b29a62e","Type":"ContainerStarted","Data":"8457d7c1d519336f98c697a417102107aba9bd8fb3294eba6bc52da825b685f4"}
Jul 06 15:49:30.553214 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:30.552881    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 15:49:30.572074 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:30.572030    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-94f47f95b-4wh4b" podStartSLOduration=1.662274076 podStartE2EDuration="3.572014976s" totalImagesPullingTime="1.9097409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:49:30.570107796 +0000 UTC m=+937.440154964" watchObservedRunningTime="2026-07-06 15:49:30.572014976 +0000 UTC m=+937.442062144"
Jul 06 15:49:41.563434 ip-10-0-138-21 kubenswrapper[2567]: I0706 15:49:41.563401    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94f47f95b-4wh4b"
Jul 06 16:01:30.431901 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.431871    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:30.435557 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.435538    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.437918 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.437897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cf784041-openshift-default-dockercfg-6c8hl\""
Jul 06 16:01:30.448546 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.448519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:30.560719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.560719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.560967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.560967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.560967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.560967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.561143 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.560977    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpz8x\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.561143 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.561021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.561143 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.561053    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661847 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661847 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661847 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661759    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661847 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.661847 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662116 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662116 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662116 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.661974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpz8x\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662116 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.662082    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662319 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.662219    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662319 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.662242    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662426 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.662395    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.662614 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.662588    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.664487 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.664458    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.664583 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.664566    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.685742 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.685648    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.686152 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.686134    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpz8x\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x\") pod \"e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") " pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.746243 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.746213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:30.891030 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.891001    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:30.896750 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.896716    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:30.896848 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.896782    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:30.896848 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.896811    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:30.920484 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:30.920457    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" event={"ID":"0cafc033-26a6-4fe2-8dae-514ca28fc5f6","Type":"ContainerStarted","Data":"e2f0be2edfac1ed5267b6196cc0dfd00a3c78b016e1347f67c4ec2562359f067"}
Jul 06 16:01:31.765170 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.765134    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:31.769400 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.769381    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.771717 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.771693    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-cf784041-serving-cert\""
Jul 06 16:01:31.778709 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.778253    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:31.872714 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.872643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfzh\" (UniqueName: \"kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.872871 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.872737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.872871 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.872785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.926322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.926286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" event={"ID":"0cafc033-26a6-4fe2-8dae-514ca28fc5f6","Type":"ContainerStarted","Data":"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"}
Jul 06 16:01:31.953877 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.953821    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" podStartSLOduration=1.953802888 podStartE2EDuration="1.953802888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:31.951830314 +0000 UTC m=+1658.821877483" watchObservedRunningTime="2026-07-06 16:01:31.953802888 +0000 UTC m=+1658.823850055"
Jul 06 16:01:31.973787 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.973754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfzh\" (UniqueName: \"kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.973951 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.973833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.974122 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.974021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.974739 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.974721    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.976494 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.976470    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:31.982371 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:31.982350    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfzh\" (UniqueName: \"kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh\") pod \"maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") " pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:32.082928 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.082842    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:32.237251 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.237220    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:32.237540 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:01:32.237519    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1abaae_359e_4b18_ab34_806d8cf97106.slice/crio-ebd7368940029e7c6c151d3682e307f9f853f54f4d93e09843a09ae21116e830 WatchSource:0}: Error finding container ebd7368940029e7c6c151d3682e307f9f853f54f4d93e09843a09ae21116e830: Status 404 returned error can't find the container with id ebd7368940029e7c6c151d3682e307f9f853f54f4d93e09843a09ae21116e830
Jul 06 16:01:32.239008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.238988    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:01:32.746821 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.746801    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:32.751891 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.751862    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:32.933141 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.933095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" event={"ID":"0b1abaae-359e-4b18-ab34-806d8cf97106","Type":"ContainerStarted","Data":"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"}
Jul 06 16:01:32.933585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.933153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" event={"ID":"0b1abaae-359e-4b18-ab34-806d8cf97106","Type":"ContainerStarted","Data":"ebd7368940029e7c6c151d3682e307f9f853f54f4d93e09843a09ae21116e830"}
Jul 06 16:01:32.933585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.933303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:32.933585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.933349    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:32.934365 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.934349    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:32.953863 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:32.953809    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" podStartSLOduration=1.501732083 podStartE2EDuration="1.953793073s" totalImagesPullingTime="452.06099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:32.952587306 +0000 UTC m=+1659.822634499" watchObservedRunningTime="2026-07-06 16:01:32.953793073 +0000 UTC m=+1659.823840241"
Jul 06 16:01:36.943583 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:36.943539    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:36.944105 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:36.943797    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" podUID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" containerName="istio-proxy" containerID="cri-o://87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59" gracePeriod=30
Jul 06 16:01:37.220813 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.220728    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:37.220976 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.220917    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" podUID="0b1abaae-359e-4b18-ab34-806d8cf97106" containerName="maas-api" containerID="cri-o://95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa" gracePeriod=30
Jul 06 16:01:37.226309 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.226257    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:37.475112 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.475036    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:37.624607 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.624570    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcfzh\" (UniqueName: \"kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh\") pod \"0b1abaae-359e-4b18-ab34-806d8cf97106\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") "
Jul 06 16:01:37.624607 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.624608    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls\") pod \"0b1abaae-359e-4b18-ab34-806d8cf97106\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") "
Jul 06 16:01:37.624853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.624750    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca\") pod \"0b1abaae-359e-4b18-ab34-806d8cf97106\" (UID: \"0b1abaae-359e-4b18-ab34-806d8cf97106\") "
Jul 06 16:01:37.625117 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.625091    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b1abaae-359e-4b18-ab34-806d8cf97106" (UID: "0b1abaae-359e-4b18-ab34-806d8cf97106"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:01:37.626875 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.626851    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b1abaae-359e-4b18-ab34-806d8cf97106" (UID: "0b1abaae-359e-4b18-ab34-806d8cf97106"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:01:37.626875 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.626863    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh" (OuterVolumeSpecName: "kube-api-access-rcfzh") pod "0b1abaae-359e-4b18-ab34-806d8cf97106" (UID: "0b1abaae-359e-4b18-ab34-806d8cf97106"). InnerVolumeSpecName "kube-api-access-rcfzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:01:37.725696 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.725589    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abaae-359e-4b18-ab34-806d8cf97106-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:37.725696 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.725618    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfzh\" (UniqueName: \"kubernetes.io/projected/0b1abaae-359e-4b18-ab34-806d8cf97106-kube-api-access-rcfzh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:37.725696 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.725628    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1abaae-359e-4b18-ab34-806d8cf97106-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:37.958711 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.958654    2567 generic.go:356] "Generic (PLEG): container finished" podID="0b1abaae-359e-4b18-ab34-806d8cf97106" containerID="95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa" exitCode=0
Jul 06 16:01:37.958711 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.958697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" event={"ID":"0b1abaae-359e-4b18-ab34-806d8cf97106","Type":"ContainerDied","Data":"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"}
Jul 06 16:01:37.959233 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.958724    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"
Jul 06 16:01:37.959233 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.958752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b" event={"ID":"0b1abaae-359e-4b18-ab34-806d8cf97106","Type":"ContainerDied","Data":"ebd7368940029e7c6c151d3682e307f9f853f54f4d93e09843a09ae21116e830"}
Jul 06 16:01:37.959233 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.958773    2567 scope.go:117] "RemoveContainer" containerID="95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"
Jul 06 16:01:37.969658 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.969640    2567 scope.go:117] "RemoveContainer" containerID="95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"
Jul 06 16:01:37.969944 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:01:37.969924    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa\": container with ID starting with 95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa not found: ID does not exist" containerID="95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"
Jul 06 16:01:37.969994 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.969953    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa"} err="failed to get container status \"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa\": rpc error: code = NotFound desc = could not find container \"95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa\": container with ID starting with 95e9a9f1b89a5cd317137beea58d62f9d07a335c27beb9ec01307f93972144fa not found: ID does not exist"
Jul 06 16:01:37.978930 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.978879    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:37.982482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:37.982460    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-cf784041-64b4ff74d8-t9t4b"]
Jul 06 16:01:39.664267 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:39.664238    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1abaae-359e-4b18-ab34-806d8cf97106" path="/var/lib/kubelet/pods/0b1abaae-359e-4b18-ab34-806d8cf97106/volumes"
Jul 06 16:01:42.200634 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.200611    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:42.267974 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.267890    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.267974 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.267931    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.267974 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.267951    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.267994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.268017    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.268061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.268094    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.268115    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpz8x\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.268179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.268170    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert\") pod \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\" (UID: \"0cafc033-26a6-4fe2-8dae-514ca28fc5f6\") "
Jul 06 16:01:42.269953 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.269919    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:01:42.271721 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.271639    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:01:42.271900 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.271827    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:01:42.272014 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.271982    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:01:42.272014 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.271985    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:01:42.272143 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.272020    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:01:42.272224 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.272178    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:01:42.272279 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.272254    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:01:42.273518 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.273489    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x" (OuterVolumeSpecName: "kube-api-access-jpz8x") pod "0cafc033-26a6-4fe2-8dae-514ca28fc5f6" (UID: "0cafc033-26a6-4fe2-8dae-514ca28fc5f6"). InnerVolumeSpecName "kube-api-access-jpz8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:01:42.369736 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369701    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369736 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369733    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369736 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369742    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369750    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369759    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369766    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369774    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369782    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.369967 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.369790    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpz8x\" (UniqueName: \"kubernetes.io/projected/0cafc033-26a6-4fe2-8dae-514ca28fc5f6-kube-api-access-jpz8x\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:42.983360 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.983321    2567 generic.go:356] "Generic (PLEG): container finished" podID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" containerID="87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59" exitCode=0
Jul 06 16:01:42.983544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.983377    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"
Jul 06 16:01:42.983544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.983376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" event={"ID":"0cafc033-26a6-4fe2-8dae-514ca28fc5f6","Type":"ContainerDied","Data":"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"}
Jul 06 16:01:42.983544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.983500    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w" event={"ID":"0cafc033-26a6-4fe2-8dae-514ca28fc5f6","Type":"ContainerDied","Data":"e2f0be2edfac1ed5267b6196cc0dfd00a3c78b016e1347f67c4ec2562359f067"}
Jul 06 16:01:42.983544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.983525    2567 scope.go:117] "RemoveContainer" containerID="87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"
Jul 06 16:01:42.992812 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.992792    2567 scope.go:117] "RemoveContainer" containerID="87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"
Jul 06 16:01:42.993137 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:01:42.993115    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59\": container with ID starting with 87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59 not found: ID does not exist" containerID="87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"
Jul 06 16:01:42.993184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:42.993148    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59"} err="failed to get container status \"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59\": rpc error: code = NotFound desc = could not find container \"87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59\": container with ID starting with 87aab2be34967d0cf58f0090549754f5ce9c54e7714fbc4eb2aadd2dfa7c2a59 not found: ID does not exist"
Jul 06 16:01:43.009725 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.006289    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:43.013277 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.013250    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cf784041-openshift-default-54496ddddf-6sr5w"]
Jul 06 16:01:43.530853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.530822    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:01:43.531324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531290    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" containerName="istio-proxy"
Jul 06 16:01:43.531324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531308    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" containerName="istio-proxy"
Jul 06 16:01:43.531324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531320    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1abaae-359e-4b18-ab34-806d8cf97106" containerName="maas-api"
Jul 06 16:01:43.531324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531328    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1abaae-359e-4b18-ab34-806d8cf97106" containerName="maas-api"
Jul 06 16:01:43.531540 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531396    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" containerName="istio-proxy"
Jul 06 16:01:43.531540 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.531405    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1abaae-359e-4b18-ab34-806d8cf97106" containerName="maas-api"
Jul 06 16:01:43.534538 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.534514    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.536838 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.536818    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7b15fef1-openshift-default-dockercfg-q2lhg\""
Jul 06 16:01:43.546391 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.546361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:01:43.665176 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.665144    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cafc033-26a6-4fe2-8dae-514ca28fc5f6" path="/var/lib/kubelet/pods/0cafc033-26a6-4fe2-8dae-514ca28fc5f6/volumes"
Jul 06 16:01:43.682482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682483    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682703 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682703 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb42w\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682703 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682703 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682833 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682833 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.682898 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.682838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.783824 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.783732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.783974 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.783915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.783974 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.783949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784115 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784238 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784190    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784298 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784298 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784397 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784397 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb42w\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784492 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784569 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784540    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784631 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784602    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.784737 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.784714    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.785077 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.785054    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.786604 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.786581    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.786604 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.786596    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.792088 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.792062    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.792315 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.792295    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb42w\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w\") pod \"e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") " pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.848569 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.848528    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:43.991345 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.991323    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:01:43.993267 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:01:43.993243    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bab783_b75e_460c_9e24_c4c06eb0a8e4.slice/crio-f92bf25a21cf2cddb94b426eea0881f39ba6da231be4f12326e757cff4d9bf2b WatchSource:0}: Error finding container f92bf25a21cf2cddb94b426eea0881f39ba6da231be4f12326e757cff4d9bf2b: Status 404 returned error can't find the container with id f92bf25a21cf2cddb94b426eea0881f39ba6da231be4f12326e757cff4d9bf2b
Jul 06 16:01:43.995724 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.995692    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:43.995805 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.995756    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:43.995805 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:43.995788    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:44.834990 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.834953    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:44.839186 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.839162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:44.841569 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.841544    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-7b15fef1-serving-cert\""
Jul 06 16:01:44.848079 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.848058    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:44.994708 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.994653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shc4d\" (UniqueName: \"kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:44.994897 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.994751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:44.994897 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.994825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:44.995716 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.995693    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerStarted","Data":"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"}
Jul 06 16:01:44.995817 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:44.995739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerStarted","Data":"f92bf25a21cf2cddb94b426eea0881f39ba6da231be4f12326e757cff4d9bf2b"}
Jul 06 16:01:45.017283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.017235    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" podStartSLOduration=2.017215808 podStartE2EDuration="2.017215808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:45.016371454 +0000 UTC m=+1671.886418621" watchObservedRunningTime="2026-07-06 16:01:45.017215808 +0000 UTC m=+1671.887262987"
Jul 06 16:01:45.096273 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.096187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shc4d\" (UniqueName: \"kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.096273 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.096239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.096509 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.096353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.097076 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.097006    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.098809 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.098784    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.107952 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.107926    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shc4d\" (UniqueName: \"kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d\") pod \"maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") " pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.151267 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.151234    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:45.295644 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.295619    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:45.297492 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:01:45.297463    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f46e4_f9dc_426d_af92_327da0fbfc21.slice/crio-592f4bf18703de823355293279f440620901aa4410e630767e5a3c998142dd2e WatchSource:0}: Error finding container 592f4bf18703de823355293279f440620901aa4410e630767e5a3c998142dd2e: Status 404 returned error can't find the container with id 592f4bf18703de823355293279f440620901aa4410e630767e5a3c998142dd2e
Jul 06 16:01:45.849057 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.849025    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:45.854482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:45.854454    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:46.002775 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:46.002739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" event={"ID":"8e5f46e4-f9dc-426d-af92-327da0fbfc21","Type":"ContainerStarted","Data":"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"}
Jul 06 16:01:46.002955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:46.002803    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" event={"ID":"8e5f46e4-f9dc-426d-af92-327da0fbfc21","Type":"ContainerStarted","Data":"592f4bf18703de823355293279f440620901aa4410e630767e5a3c998142dd2e"}
Jul 06 16:01:46.002955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:46.002843    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:46.003811 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:46.003791    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:46.078030 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:46.077913    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" podStartSLOduration=1.605934099 podStartE2EDuration="2.077893278s" totalImagesPullingTime="471.959179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:46.038326282 +0000 UTC m=+1672.908373453" watchObservedRunningTime="2026-07-06 16:01:46.077893278 +0000 UTC m=+1672.947940507"
Jul 06 16:01:47.008131 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:47.008097    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:50.382850 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.382816    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:50.383617 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.383584    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" podUID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" containerName="maas-api" containerID="cri-o://8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce" gracePeriod=30
Jul 06 16:01:50.403902 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.403866    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:50.659779 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.659749    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:50.748569 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.748531    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shc4d\" (UniqueName: \"kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d\") pod \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") "
Jul 06 16:01:50.748756 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.748621    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca\") pod \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") "
Jul 06 16:01:50.748756 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.748651    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls\") pod \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\" (UID: \"8e5f46e4-f9dc-426d-af92-327da0fbfc21\") "
Jul 06 16:01:50.749007 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.748962    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e5f46e4-f9dc-426d-af92-327da0fbfc21" (UID: "8e5f46e4-f9dc-426d-af92-327da0fbfc21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:01:50.750921 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.750895    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e5f46e4-f9dc-426d-af92-327da0fbfc21" (UID: "8e5f46e4-f9dc-426d-af92-327da0fbfc21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:01:50.751012 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.750947    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d" (OuterVolumeSpecName: "kube-api-access-shc4d") pod "8e5f46e4-f9dc-426d-af92-327da0fbfc21" (UID: "8e5f46e4-f9dc-426d-af92-327da0fbfc21"). InnerVolumeSpecName "kube-api-access-shc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:01:50.850325 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.850282    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shc4d\" (UniqueName: \"kubernetes.io/projected/8e5f46e4-f9dc-426d-af92-327da0fbfc21-kube-api-access-shc4d\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:50.850325 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.850318    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f46e4-f9dc-426d-af92-327da0fbfc21-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:50.850325 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:50.850328    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f46e4-f9dc-426d-af92-327da0fbfc21-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:01:51.026923 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.026886    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" containerID="8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce" exitCode=0
Jul 06 16:01:51.027095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.026938    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"
Jul 06 16:01:51.027095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.026974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" event={"ID":"8e5f46e4-f9dc-426d-af92-327da0fbfc21","Type":"ContainerDied","Data":"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"}
Jul 06 16:01:51.027095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.027028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47" event={"ID":"8e5f46e4-f9dc-426d-af92-327da0fbfc21","Type":"ContainerDied","Data":"592f4bf18703de823355293279f440620901aa4410e630767e5a3c998142dd2e"}
Jul 06 16:01:51.027095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.027050    2567 scope.go:117] "RemoveContainer" containerID="8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"
Jul 06 16:01:51.036290 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.036267    2567 scope.go:117] "RemoveContainer" containerID="8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"
Jul 06 16:01:51.036534 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:01:51.036515    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce\": container with ID starting with 8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce not found: ID does not exist" containerID="8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"
Jul 06 16:01:51.036574 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.036544    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce"} err="failed to get container status \"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce\": rpc error: code = NotFound desc = could not find container \"8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce\": container with ID starting with 8273db7a53f46c43e028f72fb0a0868b53dd26b19b98a5b5d6df8946ab6ee5ce not found: ID does not exist"
Jul 06 16:01:51.051184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.051161    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:51.053571 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.053546    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-7b15fef1-58ccbcd454-ntl47"]
Jul 06 16:01:51.664414 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:51.664380    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" path="/var/lib/kubelet/pods/8e5f46e4-f9dc-426d-af92-327da0fbfc21/volumes"
Jul 06 16:01:55.049220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.049121    2567 generic.go:356] "Generic (PLEG): container finished" podID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerID="7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf" exitCode=0
Jul 06 16:01:55.049220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.049193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerDied","Data":"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"}
Jul 06 16:01:55.049865 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.049612    2567 scope.go:117] "RemoveContainer" containerID="7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"
Jul 06 16:01:55.050719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.050687    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:55.050818 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.050759    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:55.050818 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:55.050794    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:01:56.056167 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:56.056131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerStarted","Data":"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"}
Jul 06 16:01:56.261949 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:56.261912    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:01:56.849711 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:56.849659    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:56.854298 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:56.854266    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:57.061408 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:57.061340    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy" containerID="cri-o://213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670" gracePeriod=30
Jul 06 16:01:57.061895 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:57.061411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:01:57.062523 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:01:57.062504    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:02:02.323040 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.322982    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:02:02.461831 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.461796    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.461853    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.461880    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.461901    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.461962    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462010    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb42w\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462062    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462091    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:02.462266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462215    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket\") pod \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\" (UID: \"25bab783-b75e-460c-9e24-c4c06eb0a8e4\") "
Jul 06 16:02:02.462266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462225    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:02.462521 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462409    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:02:02.462577 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462533    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.462577 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462549    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.462577 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462565    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.462730 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462650    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:02.462730 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.462718    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:02.464781 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.464756    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:02.464889 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.464781    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:02:02.464938 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.464888    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:02:02.464938 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.464892    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w" (OuterVolumeSpecName: "kube-api-access-nb42w") pod "25bab783-b75e-460c-9e24-c4c06eb0a8e4" (UID: "25bab783-b75e-460c-9e24-c4c06eb0a8e4"). InnerVolumeSpecName "kube-api-access-nb42w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:02:02.563718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563650    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.563718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563711    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb42w\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-kube-api-access-nb42w\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.563718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563724    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.563718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563732    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.564000 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563740    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.564000 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.563748    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bab783-b75e-460c-9e24-c4c06eb0a8e4-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:02.734140 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734105    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:02.734561 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734541    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" containerName="maas-api"
Jul 06 16:02:02.734654 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734564    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" containerName="maas-api"
Jul 06 16:02:02.734654 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734586    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:02.734654 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734594    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:02.734870 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734724    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:02.734870 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734739    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:02.734870 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.734749    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5f46e4-f9dc-426d-af92-327da0fbfc21" containerName="maas-api"
Jul 06 16:02:02.737313 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.737292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.739966 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.739782    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-85bbf8d8-openshift-default-dockercfg-n74wq\""
Jul 06 16:02:02.753011 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.752990    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:02.866520 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866722 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866722 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866722 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866851 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866851 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866851 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866851 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.866987 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.866892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zv7\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967415 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zv7\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967415 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967625    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.967718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.968057 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967916    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.968057 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.967938    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.968057 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.968026    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.968209 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.968191    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.968423 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.968406    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.970010 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.969993    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.970088 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.970072    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.976879 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.976857    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:02.977102 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:02.977084    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zv7\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7\") pod \"e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") " pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:03.049053 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.048981    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:03.093505 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.093468    2567 generic.go:356] "Generic (PLEG): container finished" podID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerID="213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670" exitCode=0
Jul 06 16:02:03.093705 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.093519    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"
Jul 06 16:02:03.093705 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.093550    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerDied","Data":"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"}
Jul 06 16:02:03.093705 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.093616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m" event={"ID":"25bab783-b75e-460c-9e24-c4c06eb0a8e4","Type":"ContainerDied","Data":"f92bf25a21cf2cddb94b426eea0881f39ba6da231be4f12326e757cff4d9bf2b"}
Jul 06 16:02:03.093705 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.093649    2567 scope.go:117] "RemoveContainer" containerID="213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"
Jul 06 16:02:03.103282 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.103202    2567 scope.go:117] "RemoveContainer" containerID="7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"
Jul 06 16:02:03.115604 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.115586    2567 scope.go:117] "RemoveContainer" containerID="213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"
Jul 06 16:02:03.115923 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:03.115893    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670\": container with ID starting with 213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670 not found: ID does not exist" containerID="213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"
Jul 06 16:02:03.116047 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.115933    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670"} err="failed to get container status \"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670\": rpc error: code = NotFound desc = could not find container \"213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670\": container with ID starting with 213565728f954e9b4413748e380c5ba1f73671db75ffc72fb605798e2dab6670 not found: ID does not exist"
Jul 06 16:02:03.116047 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.115960    2567 scope.go:117] "RemoveContainer" containerID="7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"
Jul 06 16:02:03.116263 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:03.116241    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf\": container with ID starting with 7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf not found: ID does not exist" containerID="7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"
Jul 06 16:02:03.116341 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.116272    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf"} err="failed to get container status \"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf\": rpc error: code = NotFound desc = could not find container \"7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf\": container with ID starting with 7ea16feb68f34825f0d4f4de7f19256c7c8fa1c97750c06f9c03ac5df10ce5cf not found: ID does not exist"
Jul 06 16:02:03.121485 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.121462    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:02:03.126353 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.126329    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7b15fef1-openshift-default-59f6f559df-28q8m"]
Jul 06 16:02:03.194756 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.194722    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:03.198267 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:02:03.198237    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42556c8_f5ae_4c6a_9f1d_1dc52d09f468.slice/crio-bb0ccc8b6a4d6592040dd9167b7792959ae7e87fdab6c3baaf2dfb6087b9b2ab WatchSource:0}: Error finding container bb0ccc8b6a4d6592040dd9167b7792959ae7e87fdab6c3baaf2dfb6087b9b2ab: Status 404 returned error can't find the container with id bb0ccc8b6a4d6592040dd9167b7792959ae7e87fdab6c3baaf2dfb6087b9b2ab
Jul 06 16:02:03.200468 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.200434    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:02:03.200549 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.200502    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:02:03.200549 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.200530    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:02:03.665598 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:03.665564    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" path="/var/lib/kubelet/pods/25bab783-b75e-460c-9e24-c4c06eb0a8e4/volumes"
Jul 06 16:02:04.046211 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.046175    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:04.046790 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.046772    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:04.046852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.046794    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bab783-b75e-460c-9e24-c4c06eb0a8e4" containerName="istio-proxy"
Jul 06 16:02:04.049707 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.049655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.052319 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.052285    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-85bbf8d8-serving-cert\""
Jul 06 16:02:04.062397 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.062374    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:04.100791 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.100753    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" event={"ID":"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468","Type":"ContainerStarted","Data":"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"}
Jul 06 16:02:04.100954 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.100806    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" event={"ID":"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468","Type":"ContainerStarted","Data":"bb0ccc8b6a4d6592040dd9167b7792959ae7e87fdab6c3baaf2dfb6087b9b2ab"}
Jul 06 16:02:04.152593 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.152529    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" podStartSLOduration=2.152505182 podStartE2EDuration="2.152505182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:02:04.150416118 +0000 UTC m=+1691.020463310" watchObservedRunningTime="2026-07-06 16:02:04.152505182 +0000 UTC m=+1691.022552427"
Jul 06 16:02:04.179977 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.179939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.180164 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.180038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.180164 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.180126    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvv6d\" (UniqueName: \"kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.281008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.280967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvv6d\" (UniqueName: \"kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.281306 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.281288    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.281519 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.281483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.281900 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:04.281866    2567 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-85bbf8d8-serving-cert: secret "maas-api-e2e-derive-85bbf8d8-serving-cert" not found
Jul 06 16:02:04.282060 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:04.282043    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls podName:4f47c263-c605-4d9c-bfe6-c3d834dee383 nodeName:}" failed. No retries permitted until 2026-07-06 16:02:04.782023424 +0000 UTC m=+1691.652070584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls") pod "maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" (UID: "4f47c263-c605-4d9c-bfe6-c3d834dee383") : secret "maas-api-e2e-derive-85bbf8d8-serving-cert" not found
Jul 06 16:02:04.282883 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.282856    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.298190 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.298139    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvv6d\" (UniqueName: \"kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.787356 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.787314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.790050 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.790021    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") pod \"maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") " pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:04.961878 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:04.961849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:05.049619 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:05.049543    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:05.057000 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:05.056970    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:05.107436 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:05.107407    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:05.108816 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:05.108793    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:05.108920 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:05.108834    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:05.111799 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:02:05.111774    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f47c263_c605_4d9c_bfe6_c3d834dee383.slice/crio-1e77ead2e295e6e3c3d6529e2237629ba63de0c746be3cf9aff529cfd7b47689 WatchSource:0}: Error finding container 1e77ead2e295e6e3c3d6529e2237629ba63de0c746be3cf9aff529cfd7b47689: Status 404 returned error can't find the container with id 1e77ead2e295e6e3c3d6529e2237629ba63de0c746be3cf9aff529cfd7b47689
Jul 06 16:02:06.114822 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:06.114776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" event={"ID":"4f47c263-c605-4d9c-bfe6-c3d834dee383","Type":"ContainerStarted","Data":"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"}
Jul 06 16:02:06.115274 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:06.114836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" event={"ID":"4f47c263-c605-4d9c-bfe6-c3d834dee383","Type":"ContainerStarted","Data":"1e77ead2e295e6e3c3d6529e2237629ba63de0c746be3cf9aff529cfd7b47689"}
Jul 06 16:02:06.115274 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:06.114966    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:06.132576 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:06.132533    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" podStartSLOduration=1.79060905 podStartE2EDuration="2.132517482s" totalImagesPullingTime="341.908432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:02:06.131211852 +0000 UTC m=+1693.001259022" watchObservedRunningTime="2026-07-06 16:02:06.132517482 +0000 UTC m=+1693.002564649"
Jul 06 16:02:08.616966 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:08.616922    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:08.617450 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:08.617202    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" podUID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" containerName="istio-proxy" containerID="cri-o://64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656" gracePeriod=30
Jul 06 16:02:08.810053 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:08.810012    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:08.810330 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:08.810252    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" podUID="4f47c263-c605-4d9c-bfe6-c3d834dee383" containerName="maas-api" containerID="cri-o://42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062" gracePeriod=30
Jul 06 16:02:09.118013 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.117989    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:09.129195 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.129167    2567 generic.go:356] "Generic (PLEG): container finished" podID="4f47c263-c605-4d9c-bfe6-c3d834dee383" containerID="42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062" exitCode=0
Jul 06 16:02:09.129310 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.129202    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"
Jul 06 16:02:09.129310 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.129253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" event={"ID":"4f47c263-c605-4d9c-bfe6-c3d834dee383","Type":"ContainerDied","Data":"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"}
Jul 06 16:02:09.129310 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.129301    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6" event={"ID":"4f47c263-c605-4d9c-bfe6-c3d834dee383","Type":"ContainerDied","Data":"1e77ead2e295e6e3c3d6529e2237629ba63de0c746be3cf9aff529cfd7b47689"}
Jul 06 16:02:09.129429 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.129314    2567 scope.go:117] "RemoveContainer" containerID="42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"
Jul 06 16:02:09.138574 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.138549    2567 scope.go:117] "RemoveContainer" containerID="42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"
Jul 06 16:02:09.138905 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:09.138878    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062\": container with ID starting with 42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062 not found: ID does not exist" containerID="42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"
Jul 06 16:02:09.139021 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.138912    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062"} err="failed to get container status \"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062\": rpc error: code = NotFound desc = could not find container \"42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062\": container with ID starting with 42a299df94033bf51589bdd7b859814c6376dd01a8375c1a012cf26140032062 not found: ID does not exist"
Jul 06 16:02:09.226978 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.226938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") pod \"4f47c263-c605-4d9c-bfe6-c3d834dee383\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") "
Jul 06 16:02:09.227184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.227000    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca\") pod \"4f47c263-c605-4d9c-bfe6-c3d834dee383\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") "
Jul 06 16:02:09.227184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.227098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvv6d\" (UniqueName: \"kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d\") pod \"4f47c263-c605-4d9c-bfe6-c3d834dee383\" (UID: \"4f47c263-c605-4d9c-bfe6-c3d834dee383\") "
Jul 06 16:02:09.227343 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.227319    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f47c263-c605-4d9c-bfe6-c3d834dee383" (UID: "4f47c263-c605-4d9c-bfe6-c3d834dee383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:02:09.229212 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.229177    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f47c263-c605-4d9c-bfe6-c3d834dee383" (UID: "4f47c263-c605-4d9c-bfe6-c3d834dee383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:02:09.229212 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.229187    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d" (OuterVolumeSpecName: "kube-api-access-nvv6d") pod "4f47c263-c605-4d9c-bfe6-c3d834dee383" (UID: "4f47c263-c605-4d9c-bfe6-c3d834dee383"). InnerVolumeSpecName "kube-api-access-nvv6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:02:09.328516 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.328482    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f47c263-c605-4d9c-bfe6-c3d834dee383-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:09.328516 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.328511    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f47c263-c605-4d9c-bfe6-c3d834dee383-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:09.328516 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.328521    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvv6d\" (UniqueName: \"kubernetes.io/projected/4f47c263-c605-4d9c-bfe6-c3d834dee383-kube-api-access-nvv6d\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:09.450681 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.450625    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:09.456775 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.456747    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-85bbf8d8-5cdbcccc8f-tl8v6"]
Jul 06 16:02:09.664886 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:09.664808    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f47c263-c605-4d9c-bfe6-c3d834dee383" path="/var/lib/kubelet/pods/4f47c263-c605-4d9c-bfe6-c3d834dee383/volumes"
Jul 06 16:02:15.579431 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.579407    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:15.682411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682363    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682388    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682428    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682466    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682482    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86zv7\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682553    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682588    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs\") pod \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\" (UID: \"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468\") "
Jul 06 16:02:15.682692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682604    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:15.683043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682863    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:15.683043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682903    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:02:15.683043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682974    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.683043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.682993    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.683219 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.683092    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:15.683772 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.683737    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data" (OuterVolumeSpecName: "istio-data") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:15.684861 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.684827    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token" (OuterVolumeSpecName: "istio-token") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:02:15.684861 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.684839    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7" (OuterVolumeSpecName: "kube-api-access-86zv7") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "kube-api-access-86zv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:02:15.685020 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.684887    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:02:15.685082 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.685035    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" (UID: "d42556c8-f5ae-4c6a-9f1d-1dc52d09f468"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:02:15.783955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783925    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.783955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783950    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zv7\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-kube-api-access-86zv7\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.783955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783960    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.784169 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783970    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.784169 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783978    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.784169 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783985    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:15.784169 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:15.783992    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:02:16.169294 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.169263    2567 generic.go:356] "Generic (PLEG): container finished" podID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" containerID="64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656" exitCode=0
Jul 06 16:02:16.169448 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.169389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" event={"ID":"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468","Type":"ContainerDied","Data":"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"}
Jul 06 16:02:16.169448 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.169428    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"
Jul 06 16:02:16.169531 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.169445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78" event={"ID":"d42556c8-f5ae-4c6a-9f1d-1dc52d09f468","Type":"ContainerDied","Data":"bb0ccc8b6a4d6592040dd9167b7792959ae7e87fdab6c3baaf2dfb6087b9b2ab"}
Jul 06 16:02:16.169531 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.169484    2567 scope.go:117] "RemoveContainer" containerID="64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"
Jul 06 16:02:16.179414 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.179389    2567 scope.go:117] "RemoveContainer" containerID="64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"
Jul 06 16:02:16.179711 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:02:16.179648    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656\": container with ID starting with 64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656 not found: ID does not exist" containerID="64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"
Jul 06 16:02:16.179789 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.179715    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656"} err="failed to get container status \"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656\": rpc error: code = NotFound desc = could not find container \"64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656\": container with ID starting with 64bb9c70e6bd52dece045cb7e5d35ef763b0c2a7a38ebc0401d7770696934656 not found: ID does not exist"
Jul 06 16:02:16.191992 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.191969    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:16.196633 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:16.196612    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-85bbf8d8-openshift-default-5576f6b5ff-rhc78"]
Jul 06 16:02:17.665572 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:02:17.665482    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" path="/var/lib/kubelet/pods/d42556c8-f5ae-4c6a-9f1d-1dc52d09f468/volumes"
Jul 06 16:03:53.069429 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069396    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:03:53.069845 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069803    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f47c263-c605-4d9c-bfe6-c3d834dee383" containerName="maas-api"
Jul 06 16:03:53.069845 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069816    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f47c263-c605-4d9c-bfe6-c3d834dee383" containerName="maas-api"
Jul 06 16:03:53.069845 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069842    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" containerName="istio-proxy"
Jul 06 16:03:53.069952 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069848    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" containerName="istio-proxy"
Jul 06 16:03:53.069952 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069903    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f47c263-c605-4d9c-bfe6-c3d834dee383" containerName="maas-api"
Jul 06 16:03:53.069952 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.069913    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42556c8-f5ae-4c6a-9f1d-1dc52d09f468" containerName="istio-proxy"
Jul 06 16:03:53.072895 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.072878    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.075418 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.075395    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-508497ba-openshift-default-dockercfg-gl6vz\""
Jul 06 16:03:53.094599 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.094576    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:03:53.151108 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151108 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151385 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151385 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151385 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151564 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjkd\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151564 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151658 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.151826 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.151809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.252875 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.252838    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.252886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.252903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.252954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.252974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253061 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjkd\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253290    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253536 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253515    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253602 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253547    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253602 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253566    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.253811 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.253794    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.255949 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.255919    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.256069 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.256052    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.262041 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.262021    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.262386 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.262365    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjkd\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd\") pod \"e2e-mt-508497ba-openshift-default-798dd964bc-qm996\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") " pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.384782 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.384683    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:53.530818 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.530771    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:03:53.533369 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:03:53.533341    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919a37a3_2f91_4e42_83fa_d5ddb791c1ef.slice/crio-0d3dc5262cb67a59e813da36e3559477ae6918ffa9b95791bb276094bbeb428c WatchSource:0}: Error finding container 0d3dc5262cb67a59e813da36e3559477ae6918ffa9b95791bb276094bbeb428c: Status 404 returned error can't find the container with id 0d3dc5262cb67a59e813da36e3559477ae6918ffa9b95791bb276094bbeb428c
Jul 06 16:03:53.535544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.535500    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:03:53.535625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.535592    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:03:53.535688 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.535633    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:03:53.634978 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.634878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerStarted","Data":"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"}
Jul 06 16:03:53.634978 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:53.634942    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerStarted","Data":"0d3dc5262cb67a59e813da36e3559477ae6918ffa9b95791bb276094bbeb428c"}
Jul 06 16:03:54.385222 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:54.385184    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:54.390428 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:54.390406    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:54.415771 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:54.415719    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podStartSLOduration=1.415704117 podStartE2EDuration="1.415704117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:03:53.679291587 +0000 UTC m=+1800.549338755" watchObservedRunningTime="2026-07-06 16:03:54.415704117 +0000 UTC m=+1801.285751350"
Jul 06 16:03:54.639765 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:54.639688    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:54.640773 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:54.640754    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:03:56.047484 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.047440    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:03:56.051218 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.051195    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.053524 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.053500    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-508497ba-serving-cert\""
Jul 06 16:03:56.070693 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.070640    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:03:56.073779 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.073742    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hm4\" (UniqueName: \"kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.074606 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.073882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.074606 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.073929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.174616 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.174584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.174867 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.174626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.174867 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.174699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hm4\" (UniqueName: \"kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.174867 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:03:56.174811    2567 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-508497ba-serving-cert: secret "maas-api-e2e-mt-508497ba-serving-cert" not found
Jul 06 16:03:56.174972 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:03:56.174878    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls podName:7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa nodeName:}" failed. No retries permitted until 2026-07-06 16:03:56.674859248 +0000 UTC m=+1803.544906395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls") pod "maas-api-e2e-mt-508497ba-7c4555577b-hj742" (UID: "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa") : secret "maas-api-e2e-mt-508497ba-serving-cert" not found
Jul 06 16:03:56.175274 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.175255    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.190095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.190063    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hm4\" (UniqueName: \"kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.679591 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.679557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.682107 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.682086    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") pod \"maas-api-e2e-mt-508497ba-7c4555577b-hj742\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") " pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:56.962144 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:56.962103    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:57.116116 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:57.116090    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:03:57.118562 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:03:57.118535    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdec7f1_aec0_4d1e_9e4e_8ce444f503fa.slice/crio-a617f64308c3b269776f98e920292687ce780fb1b6e22971ec9c7816d1db2f79 WatchSource:0}: Error finding container a617f64308c3b269776f98e920292687ce780fb1b6e22971ec9c7816d1db2f79: Status 404 returned error can't find the container with id a617f64308c3b269776f98e920292687ce780fb1b6e22971ec9c7816d1db2f79
Jul 06 16:03:57.654828 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:57.654796    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" event={"ID":"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa","Type":"ContainerStarted","Data":"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"}
Jul 06 16:03:57.654958 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:57.654860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" event={"ID":"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa","Type":"ContainerStarted","Data":"a617f64308c3b269776f98e920292687ce780fb1b6e22971ec9c7816d1db2f79"}
Jul 06 16:03:57.655034 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:57.654971    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:03:57.684840 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:03:57.684777    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" podStartSLOduration=1.215888372 podStartE2EDuration="1.684757512s" totalImagesPullingTime="468.86914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:03:57.684382349 +0000 UTC m=+1804.554429518" watchObservedRunningTime="2026-07-06 16:03:57.684757512 +0000 UTC m=+1804.554804681"
Jul 06 16:04:03.666506 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:03.666478    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:04:08.721033 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.721003    2567 generic.go:356] "Generic (PLEG): container finished" podID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerID="bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7" exitCode=0
Jul 06 16:04:08.721461 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.721079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerDied","Data":"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"}
Jul 06 16:04:08.721505 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.721491    2567 scope.go:117] "RemoveContainer" containerID="bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"
Jul 06 16:04:08.722542 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.722514    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:08.722685 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.722636    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:08.722766 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.722722    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:08.918857 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.918819    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:04:08.919057 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:08.919025    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" podUID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" containerName="maas-api" containerID="cri-o://954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1" gracePeriod=30
Jul 06 16:04:09.180955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.180933    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:04:09.298976 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.298884    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca\") pod \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") "
Jul 06 16:04:09.299138 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.298988    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4hm4\" (UniqueName: \"kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4\") pod \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") "
Jul 06 16:04:09.299138 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.299039    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") pod \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\" (UID: \"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa\") "
Jul 06 16:04:09.299263 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.299230    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" (UID: "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:04:09.301174 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.301146    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4" (OuterVolumeSpecName: "kube-api-access-s4hm4") pod "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" (UID: "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa"). InnerVolumeSpecName "kube-api-access-s4hm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:04:09.301283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.301187    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" (UID: "7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:04:09.399706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.399644    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:09.399706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.399701    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:09.399706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.399712    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4hm4\" (UniqueName: \"kubernetes.io/projected/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa-kube-api-access-s4hm4\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:09.639934 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.639837    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:09.730430 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.730386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerStarted","Data":"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"}
Jul 06 16:04:09.732325 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.732295    2567 generic.go:356] "Generic (PLEG): container finished" podID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" containerID="954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1" exitCode=0
Jul 06 16:04:09.732452 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.732368    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" event={"ID":"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa","Type":"ContainerDied","Data":"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"}
Jul 06 16:04:09.732452 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.732420    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742" event={"ID":"7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa","Type":"ContainerDied","Data":"a617f64308c3b269776f98e920292687ce780fb1b6e22971ec9c7816d1db2f79"}
Jul 06 16:04:09.732452 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.732441    2567 scope.go:117] "RemoveContainer" containerID="954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"
Jul 06 16:04:09.732641 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.732374    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"
Jul 06 16:04:09.742652 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.742632    2567 scope.go:117] "RemoveContainer" containerID="954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"
Jul 06 16:04:09.742926 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:04:09.742907    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1\": container with ID starting with 954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1 not found: ID does not exist" containerID="954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"
Jul 06 16:04:09.742985 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.742933    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1"} err="failed to get container status \"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1\": rpc error: code = NotFound desc = could not find container \"954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1\": container with ID starting with 954e9817e2cd559ecc8a6633785b93c2673ffedd8b5bd3744265c3a47e37b4f1 not found: ID does not exist"
Jul 06 16:04:09.784511 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.784441    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:04:09.787491 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:09.787453    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-508497ba-7c4555577b-hj742"]
Jul 06 16:04:10.385011 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:10.384963    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:10.386362 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:10.386317    2567 patch_prober.go:28] interesting pod/e2e-mt-508497ba-openshift-default-798dd964bc-qm996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 06 16:04:10.386509 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:10.386391    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 06 16:04:11.385344 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:11.385296    2567 patch_prober.go:28] interesting pod/e2e-mt-508497ba-openshift-default-798dd964bc-qm996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 06 16:04:11.385891 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:11.385369    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 06 16:04:11.664165 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:11.664088    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" path="/var/lib/kubelet/pods/7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa/volumes"
Jul 06 16:04:12.385955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:12.385913    2567 patch_prober.go:28] interesting pod/e2e-mt-508497ba-openshift-default-798dd964bc-qm996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 06 16:04:12.386441 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:12.385995    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 06 16:04:13.386073 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:13.385572    2567 patch_prober.go:28] interesting pod/e2e-mt-508497ba-openshift-default-798dd964bc-qm996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 06 16:04:13.386073 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:13.385641    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 06 16:04:14.389108 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:14.389078    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:14.389476 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:14.389435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:14.390069 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:14.390052    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:15.695921 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:15.695884    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:04:15.762714 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:15.762651    2567 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" secret="" err="secret \"e2e-mt-508497ba-openshift-default-dockercfg-gl6vz\" not found"
Jul 06 16:04:16.772198 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:16.772141    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy" containerID="cri-o://baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac" gracePeriod=30
Jul 06 16:04:22.033411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.033345    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:22.122004 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.121974    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122191 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122027    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122191 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122191 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122191 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122105    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vjkd\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122410 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122250    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122410 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122302    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122410 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122410 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122377    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket\") pod \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\" (UID: \"919a37a3-2f91-4e42-83fa-d5ddb791c1ef\") "
Jul 06 16:04:22.122718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122425    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:04:22.122718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122597    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:04:22.122826 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122763    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.122826 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122783    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.122826 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122805    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:04:22.122970 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.122907    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:04:22.123300 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.123279    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:04:22.124950 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.124923    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:04:22.125033 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.124948    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd" (OuterVolumeSpecName: "kube-api-access-8vjkd") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "kube-api-access-8vjkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:04:22.125033 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.125006    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:04:22.125033 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.125016    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "919a37a3-2f91-4e42-83fa-d5ddb791c1ef" (UID: "919a37a3-2f91-4e42-83fa-d5ddb791c1ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:04:22.224229 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224199    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224229 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224227    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjkd\" (UniqueName: \"kubernetes.io/projected/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-kube-api-access-8vjkd\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224229 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224238    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224453 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224246    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224453 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224254    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224453 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224261    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.224453 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.224269    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919a37a3-2f91-4e42-83fa-d5ddb791c1ef-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:04:22.807598 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.807564    2567 generic.go:356] "Generic (PLEG): container finished" podID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerID="baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac" exitCode=0
Jul 06 16:04:22.807780 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.807629    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"
Jul 06 16:04:22.807780 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.807643    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerDied","Data":"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"}
Jul 06 16:04:22.807780 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.807741    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996" event={"ID":"919a37a3-2f91-4e42-83fa-d5ddb791c1ef","Type":"ContainerDied","Data":"0d3dc5262cb67a59e813da36e3559477ae6918ffa9b95791bb276094bbeb428c"}
Jul 06 16:04:22.807780 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.807765    2567 scope.go:117] "RemoveContainer" containerID="baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"
Jul 06 16:04:22.817378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.817361    2567 scope.go:117] "RemoveContainer" containerID="bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"
Jul 06 16:04:22.826102 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.826082    2567 scope.go:117] "RemoveContainer" containerID="baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"
Jul 06 16:04:22.826345 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:04:22.826327    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac\": container with ID starting with baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac not found: ID does not exist" containerID="baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"
Jul 06 16:04:22.826400 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.826353    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac"} err="failed to get container status \"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac\": rpc error: code = NotFound desc = could not find container \"baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac\": container with ID starting with baee4aef4ac52c137ee0c0b5d554b96c00c93926aa7d5bb15d9450714702a3ac not found: ID does not exist"
Jul 06 16:04:22.826400 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.826370    2567 scope.go:117] "RemoveContainer" containerID="bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"
Jul 06 16:04:22.826577 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:04:22.826559    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7\": container with ID starting with bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7 not found: ID does not exist" containerID="bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"
Jul 06 16:04:22.826621 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.826583    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7"} err="failed to get container status \"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7\": rpc error: code = NotFound desc = could not find container \"bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7\": container with ID starting with bacd1f95bcc96976fc2423379a82cad0c155bd55ee0a193e42d1826696869ef7 not found: ID does not exist"
Jul 06 16:04:22.830059 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.830036    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:04:22.835279 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:22.835257    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-508497ba-openshift-default-798dd964bc-qm996"]
Jul 06 16:04:23.664321 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:23.664289    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" path="/var/lib/kubelet/pods/919a37a3-2f91-4e42-83fa-d5ddb791c1ef/volumes"
Jul 06 16:04:36.770536 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770506    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:04:36.770933 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770894    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:36.770933 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770907    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:36.770933 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770916    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" containerName="maas-api"
Jul 06 16:04:36.770933 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770922    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" containerName="maas-api"
Jul 06 16:04:36.771087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.770993    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:36.771087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.771001    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdec7f1-aec0-4d1e-9e4e-8ce444f503fa" containerName="maas-api"
Jul 06 16:04:36.771087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.771009    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:36.773008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.772991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.775405 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.775383    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3159bb15-openshift-default-dockercfg-jh49s\""
Jul 06 16:04:36.785705 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.785680    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:04:36.956118 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wp6d\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956118 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956127    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956175    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956247    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956378 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956572 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:36.956572 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:36.956438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057110 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057110 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wp6d\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057110 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057197    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057581 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057418    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057581 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057581 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057450    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057581 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057581 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057497    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.057798 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.057777    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.058036 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.058012    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.059825 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.059797    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.060008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.059991    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.066182 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.066161    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.066360 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.066340    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wp6d\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d\") pod \"e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") " pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.085291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.085268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:37.225284 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.225255    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:04:37.227058 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:04:37.227028    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf67da3_2e69_4492_8684_a483678225c4.slice/crio-e03b0e835d9e8b7e9116f25da5378377adb341aebe9431c19475aa106bd081d3 WatchSource:0}: Error finding container e03b0e835d9e8b7e9116f25da5378377adb341aebe9431c19475aa106bd081d3: Status 404 returned error can't find the container with id e03b0e835d9e8b7e9116f25da5378377adb341aebe9431c19475aa106bd081d3
Jul 06 16:04:37.229490 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.229449    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:37.229602 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.229543    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:37.229602 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.229593    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:37.881984 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.881949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerStarted","Data":"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"}
Jul 06 16:04:37.882377 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.882007    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerStarted","Data":"e03b0e835d9e8b7e9116f25da5378377adb341aebe9431c19475aa106bd081d3"}
Jul 06 16:04:37.908455 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:37.908404    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podStartSLOduration=1.9083893280000002 podStartE2EDuration="1.908389328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:04:37.90673194 +0000 UTC m=+1844.776779110" watchObservedRunningTime="2026-07-06 16:04:37.908389328 +0000 UTC m=+1844.778436496"
Jul 06 16:04:38.086149 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:38.086117    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:38.090945 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:38.090921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:38.887244 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:38.887205    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:38.888263 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:38.888245    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:40.022362 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.022328    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:04:40.022767 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.022753    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:40.022818 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.022768    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="919a37a3-2f91-4e42-83fa-d5ddb791c1ef" containerName="istio-proxy"
Jul 06 16:04:40.025040 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.025022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.027249 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.027226    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-3159bb15-serving-cert\""
Jul 06 16:04:40.039100 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.039075    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:04:40.185655 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.185615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.185853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.185679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.185853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.185755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zll\" (UniqueName: \"kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.287237 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.287139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.287237 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.287189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.287237 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.287235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zll\" (UniqueName: \"kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.287944 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.287920    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.289760 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.289741    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.299009 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.298986    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zll\" (UniqueName: \"kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll\") pod \"maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") " pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.335942 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.335897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:40.480054 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.480020    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:04:40.481205 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:04:40.481179    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8886dbf9_6881_47b9_99ad_15b82466160e.slice/crio-8fd734058e6467f092b3659e48c571796e6ca6c253a02894a16df5aa484d31aa WatchSource:0}: Error finding container 8fd734058e6467f092b3659e48c571796e6ca6c253a02894a16df5aa484d31aa: Status 404 returned error can't find the container with id 8fd734058e6467f092b3659e48c571796e6ca6c253a02894a16df5aa484d31aa
Jul 06 16:04:40.897477 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:40.897444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" event={"ID":"8886dbf9-6881-47b9-99ad-15b82466160e","Type":"ContainerStarted","Data":"8fd734058e6467f092b3659e48c571796e6ca6c253a02894a16df5aa484d31aa"}
Jul 06 16:04:42.909105 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:42.909044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" event={"ID":"8886dbf9-6881-47b9-99ad-15b82466160e","Type":"ContainerStarted","Data":"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"}
Jul 06 16:04:42.909544 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:42.909179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:48.920850 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:48.920817    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:04:48.943653 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:48.943598    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" podStartSLOduration=7.642188506 podStartE2EDuration="9.943580334s" totalImagesPullingTime="2.301391828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:04:42.965127654 +0000 UTC m=+1849.835174833" watchObservedRunningTime="2026-07-06 16:04:48.943580334 +0000 UTC m=+1855.813627502"
Jul 06 16:04:50.951555 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.951519    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf67da3-2e69-4492-8684-a483678225c4" containerID="ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6" exitCode=0
Jul 06 16:04:50.951999 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.951563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerDied","Data":"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"}
Jul 06 16:04:50.952045 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.952002    2567 scope.go:117] "RemoveContainer" containerID="ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"
Jul 06 16:04:50.953021 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.952991    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:50.953081 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.953060    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:50.953126 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:50.953098    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:04:51.958893 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:51.958855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerStarted","Data":"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"}
Jul 06 16:04:52.086328 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:52.086287    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:52.087573 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:52.087521    2567 patch_prober.go:28] interesting pod/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 06 16:04:52.087723 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:52.087610    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 06 16:04:53.085969 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:53.085923    2567 patch_prober.go:28] interesting pod/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 06 16:04:53.086586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:53.086005    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 06 16:04:53.887357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:53.887320    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:54.085777 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:54.085684    2567 patch_prober.go:28] interesting pod/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 06 16:04:54.086064 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:54.086027    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 06 16:04:55.086101 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:55.086063    2567 patch_prober.go:28] interesting pod/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 06 16:04:55.086746 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:55.086712    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 06 16:04:56.090045 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:56.090009    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:04:56.090892 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:04:56.090873    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:05:27.496714 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.496621    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:05:27.497144 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.496876    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" podUID="8886dbf9-6881-47b9-99ad-15b82466160e" containerName="maas-api" containerID="cri-o://b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b" gracePeriod=30
Jul 06 16:05:27.751165 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.751100    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:05:27.832154 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.832120    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9zll\" (UniqueName: \"kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll\") pod \"8886dbf9-6881-47b9-99ad-15b82466160e\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") "
Jul 06 16:05:27.832355 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.832245    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca\") pod \"8886dbf9-6881-47b9-99ad-15b82466160e\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") "
Jul 06 16:05:27.832355 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.832312    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls\") pod \"8886dbf9-6881-47b9-99ad-15b82466160e\" (UID: \"8886dbf9-6881-47b9-99ad-15b82466160e\") "
Jul 06 16:05:27.832657 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.832630    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8886dbf9-6881-47b9-99ad-15b82466160e" (UID: "8886dbf9-6881-47b9-99ad-15b82466160e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:05:27.834299 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.834275    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll" (OuterVolumeSpecName: "kube-api-access-b9zll") pod "8886dbf9-6881-47b9-99ad-15b82466160e" (UID: "8886dbf9-6881-47b9-99ad-15b82466160e"). InnerVolumeSpecName "kube-api-access-b9zll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:05:27.834408 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.834387    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8886dbf9-6881-47b9-99ad-15b82466160e" (UID: "8886dbf9-6881-47b9-99ad-15b82466160e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:05:27.933583 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.933547    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zll\" (UniqueName: \"kubernetes.io/projected/8886dbf9-6881-47b9-99ad-15b82466160e-kube-api-access-b9zll\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:05:27.933583 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.933577    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8886dbf9-6881-47b9-99ad-15b82466160e-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:05:27.933583 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:27.933587    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8886dbf9-6881-47b9-99ad-15b82466160e-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:05:28.144774 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.144684    2567 generic.go:356] "Generic (PLEG): container finished" podID="8886dbf9-6881-47b9-99ad-15b82466160e" containerID="b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b" exitCode=0
Jul 06 16:05:28.144774 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.144738    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"
Jul 06 16:05:28.144993 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.144764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" event={"ID":"8886dbf9-6881-47b9-99ad-15b82466160e","Type":"ContainerDied","Data":"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"}
Jul 06 16:05:28.144993 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.144818    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d" event={"ID":"8886dbf9-6881-47b9-99ad-15b82466160e","Type":"ContainerDied","Data":"8fd734058e6467f092b3659e48c571796e6ca6c253a02894a16df5aa484d31aa"}
Jul 06 16:05:28.144993 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.144840    2567 scope.go:117] "RemoveContainer" containerID="b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"
Jul 06 16:05:28.154550 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.154533    2567 scope.go:117] "RemoveContainer" containerID="b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"
Jul 06 16:05:28.154849 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:05:28.154827    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b\": container with ID starting with b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b not found: ID does not exist" containerID="b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"
Jul 06 16:05:28.154911 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.154858    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b"} err="failed to get container status \"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b\": rpc error: code = NotFound desc = could not find container \"b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b\": container with ID starting with b0d24c93dc332aa55040f9432fa4eff0491b877258c09ad70f79f3dd817c572b not found: ID does not exist"
Jul 06 16:05:28.167610 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.167587    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:05:28.174929 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:28.174911    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-3159bb15-556fb49bdd-sd92d"]
Jul 06 16:05:29.664586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:05:29.664547    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8886dbf9-6881-47b9-99ad-15b82466160e" path="/var/lib/kubelet/pods/8886dbf9-6881-47b9-99ad-15b82466160e/volumes"
Jul 06 16:06:32.008484 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:32.008448    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:06:32.008959 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:32.008733    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy" containerID="cri-o://6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce" gracePeriod=30
Jul 06 16:06:37.269070 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.269007    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:06:37.375038 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.374998    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375038 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375036    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375342 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375058    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375342 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375176    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375342 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375245    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375342 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wp6d\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375342 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375323    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375600 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375353    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.375600 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375549    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:06:37.375751 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375622    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:06:37.375817 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375771    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:06:37.375817 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375626    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy\") pod \"3bf67da3-2e69-4492-8684-a483678225c4\" (UID: \"3bf67da3-2e69-4492-8684-a483678225c4\") "
Jul 06 16:06:37.376040 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.375827    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:06:37.376141 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376056    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:06:37.376203 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376160    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.376203 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376178    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.376203 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376192    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf67da3-2e69-4492-8684-a483678225c4-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.376361 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376206    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.376361 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.376219    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.377706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.377649    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:06:37.377706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.377656    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d" (OuterVolumeSpecName: "kube-api-access-8wp6d") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "kube-api-access-8wp6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:06:37.377869 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.377722    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:06:37.377949 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.377927    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bf67da3-2e69-4492-8684-a483678225c4" (UID: "3bf67da3-2e69-4492-8684-a483678225c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:06:37.476719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.476658    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf67da3-2e69-4492-8684-a483678225c4-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.476719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.476708    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.476719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.476716    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf67da3-2e69-4492-8684-a483678225c4-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.476719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.476724    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wp6d\" (UniqueName: \"kubernetes.io/projected/3bf67da3-2e69-4492-8684-a483678225c4-kube-api-access-8wp6d\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:06:37.482228 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.482197    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf67da3-2e69-4492-8684-a483678225c4" containerID="6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce" exitCode=0
Jul 06 16:06:37.482379 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.482239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerDied","Data":"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"}
Jul 06 16:06:37.482379 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.482298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496" event={"ID":"3bf67da3-2e69-4492-8684-a483678225c4","Type":"ContainerDied","Data":"e03b0e835d9e8b7e9116f25da5378377adb341aebe9431c19475aa106bd081d3"}
Jul 06 16:06:37.482379 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.482323    2567 scope.go:117] "RemoveContainer" containerID="6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"
Jul 06 16:06:37.482379 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.482250    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"
Jul 06 16:06:37.492362 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.492345    2567 scope.go:117] "RemoveContainer" containerID="ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"
Jul 06 16:06:37.500537 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.500519    2567 scope.go:117] "RemoveContainer" containerID="6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"
Jul 06 16:06:37.500812 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:06:37.500795    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce\": container with ID starting with 6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce not found: ID does not exist" containerID="6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"
Jul 06 16:06:37.500858 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.500821    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce"} err="failed to get container status \"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce\": rpc error: code = NotFound desc = could not find container \"6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce\": container with ID starting with 6b38cb81d0b912f0bd8482d6d9c88b66fffa79fd8ca0e87147033a6c7b9220ce not found: ID does not exist"
Jul 06 16:06:37.500858 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.500839    2567 scope.go:117] "RemoveContainer" containerID="ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"
Jul 06 16:06:37.501079 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:06:37.501065    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6\": container with ID starting with ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6 not found: ID does not exist" containerID="ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"
Jul 06 16:06:37.501115 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.501083    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6"} err="failed to get container status \"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6\": rpc error: code = NotFound desc = could not find container \"ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6\": container with ID starting with ac6d9bb78aca84c1394a1f5286318b2b2d78c43c12a6bc8b2d6abd82c92557a6 not found: ID does not exist"
Jul 06 16:06:37.507912 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.507880    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:06:37.515089 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.515070    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3159bb15-openshift-default-7b7667bf5f-j7496"]
Jul 06 16:06:37.666936 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:06:37.666855    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf67da3-2e69-4492-8684-a483678225c4" path="/var/lib/kubelet/pods/3bf67da3-2e69-4492-8684-a483678225c4/volumes"
Jul 06 16:07:29.547068 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547035    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547400    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547411    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547426    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547431    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547442    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8886dbf9-6881-47b9-99ad-15b82466160e" containerName="maas-api"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547448    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8886dbf9-6881-47b9-99ad-15b82466160e" containerName="maas-api"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547517    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8886dbf9-6881-47b9-99ad-15b82466160e" containerName="maas-api"
Jul 06 16:07:29.547596 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.547525    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:29.550728 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.550712    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.554146 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.554126    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-afeb72-openshift-default-dockercfg-bk6c7\""
Jul 06 16:07:29.565362 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.565333    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:07:29.736319 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736487 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736487 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736376    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736487 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736487 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjf6\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.736625 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.736575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjf6\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837789 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837789 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837789 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837655    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.837789 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.837721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.838056 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.838030    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.838140 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.838103    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.838480 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.838359    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.838561 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.838516    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.839834 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.839813    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.840109 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.840091    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.848478 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.848455    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjf6\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.848845 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.848823    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token\") pod \"e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") " pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:29.861711 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:29.861691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:30.004217 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.004106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:07:30.006412 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:07:30.006386    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab80143_a705_4cb5_bb4e_a2e0add08857.slice/crio-2db79c4b92b3595f568a64e24f799a014e8fab7c53132a1a696a394836c81dad WatchSource:0}: Error finding container 2db79c4b92b3595f568a64e24f799a014e8fab7c53132a1a696a394836c81dad: Status 404 returned error can't find the container with id 2db79c4b92b3595f568a64e24f799a014e8fab7c53132a1a696a394836c81dad
Jul 06 16:07:30.008438 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.008405    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:30.008542 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.008471    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:30.008542 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.008499    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:30.724651 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.724611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerStarted","Data":"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"}
Jul 06 16:07:30.725148 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.724692    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerStarted","Data":"2db79c4b92b3595f568a64e24f799a014e8fab7c53132a1a696a394836c81dad"}
Jul 06 16:07:30.748904 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.748859    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podStartSLOduration=1.748840411 podStartE2EDuration="1.748840411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:07:30.747201797 +0000 UTC m=+2017.617248985" watchObservedRunningTime="2026-07-06 16:07:30.748840411 +0000 UTC m=+2017.618887580"
Jul 06 16:07:30.862127 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:30.862008    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:31.867689 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:31.867639    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:32.734619 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.734585    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:32.735742 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.735723    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:32.935607 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.935573    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:07:32.936202 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.936184    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf67da3-2e69-4492-8684-a483678225c4" containerName="istio-proxy"
Jul 06 16:07:32.939484 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.939464    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:32.943332 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.943305    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-afeb72-serving-cert\""
Jul 06 16:07:32.954377 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:32.954350    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:07:33.067486 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.067392    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjctx\" (UniqueName: \"kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.067486 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.067451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.067691 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.067598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.168176 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.168141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.168176 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.168179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjctx\" (UniqueName: \"kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.168391 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.168208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.168391 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:07:33.168302    2567 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-afeb72-serving-cert: secret "maas-api-e2e-inf-a-afeb72-serving-cert" not found
Jul 06 16:07:33.168391 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:07:33.168365    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls podName:8237656f-3a4a-4db7-80fa-fa78b66296cc nodeName:}" failed. No retries permitted until 2026-07-06 16:07:33.668348798 +0000 UTC m=+2020.538395944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls") pod "maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" (UID: "8237656f-3a4a-4db7-80fa-fa78b66296cc") : secret "maas-api-e2e-inf-a-afeb72-serving-cert" not found
Jul 06 16:07:33.168924 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.168904    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.179309 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.179283    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjctx\" (UniqueName: \"kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.673769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.673736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.676418 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.676382    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") " pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.853015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.852976    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:33.996448 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.996419    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:07:33.998098 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:07:33.998068    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8237656f_3a4a_4db7_80fa_fa78b66296cc.slice/crio-e40fac1c4882e01ee469dc687ebf59b9acbbc59f50d805ea416f0dc2d7574833 WatchSource:0}: Error finding container e40fac1c4882e01ee469dc687ebf59b9acbbc59f50d805ea416f0dc2d7574833: Status 404 returned error can't find the container with id e40fac1c4882e01ee469dc687ebf59b9acbbc59f50d805ea416f0dc2d7574833
Jul 06 16:07:33.999635 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:33.999618    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:07:34.746432 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:34.746391    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" event={"ID":"8237656f-3a4a-4db7-80fa-fa78b66296cc","Type":"ContainerStarted","Data":"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"}
Jul 06 16:07:34.746714 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:34.746548    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:34.746714 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:34.746595    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" event={"ID":"8237656f-3a4a-4db7-80fa-fa78b66296cc","Type":"ContainerStarted","Data":"e40fac1c4882e01ee469dc687ebf59b9acbbc59f50d805ea416f0dc2d7574833"}
Jul 06 16:07:34.769966 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:34.769880    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" podStartSLOduration=2.261106722 podStartE2EDuration="2.769864207s" totalImagesPullingTime="508.757485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:07:34.76712004 +0000 UTC m=+2021.637167210" watchObservedRunningTime="2026-07-06 16:07:34.769864207 +0000 UTC m=+2021.639911374"
Jul 06 16:07:40.756263 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:40.756229    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:07:47.736889 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.736851    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 16:07:47.737528 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.736910    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 16:07:47.820574 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.820543    2567 generic.go:356] "Generic (PLEG): container finished" podID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerID="d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da" exitCode=0
Jul 06 16:07:47.820759 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.820616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerDied","Data":"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"}
Jul 06 16:07:47.821035 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.821019    2567 scope.go:117] "RemoveContainer" containerID="d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"
Jul 06 16:07:47.822017 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.821987    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:47.822120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.822090    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:47.822182 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:47.822143    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:07:48.827462 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:48.827432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerStarted","Data":"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"}
Jul 06 16:07:48.862588 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:48.862558    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:48.862765 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:48.862750    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 16:07:48.862821 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:48.862802    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 16:07:49.862868 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:49.862836    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 16:07:49.863248 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:49.862900    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 16:07:50.862961 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:50.862928    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 16:07:50.863363 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:50.862979    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 16:07:51.862552 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:51.862522    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 16:07:51.862753 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:51.862577    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 16:07:52.866073 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:52.866041    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:52.866442 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:52.866293    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:07:52.866888 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:07:52.866869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:08:21.071866 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.071787    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:21.074739 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.074719    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.084588 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.084568    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-785a99\"/\"kube-root-ca.crt\""
Jul 06 16:08:21.085555 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.085527    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-785a99\"/\"test-model-785a99-kserve-self-signed-certs\""
Jul 06 16:08:21.085630 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.085576    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-785a99\"/\"default-dockercfg-6v26j\""
Jul 06 16:08:21.085825 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.085812    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-785a99\"/\"openshift-service-ca.crt\""
Jul 06 16:08:21.099146 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.099123    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:21.107557 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.107713 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.107713 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107608    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.107853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107714    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.107853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.107853 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.107825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786nr\" (UniqueName: \"kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209115 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786nr\" (UniqueName: \"kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209283 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209499 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209499 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209476    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209617 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209549    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.209754 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.209738    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.211421 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.211399    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.211783 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.211764    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.222043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.222019    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786nr\" (UniqueName: \"kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr\") pod \"test-model-785a99-kserve-5c99496cc9-fxl85\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") " pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.383982 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.383882    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:21.542316 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:08:21.542285    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod249d4ff4_c4e4_4244_aedd_15d378b98786.slice/crio-caa459baea479464f64201d130df9e5c838bc0c2fb3129cd3edbfb5d8cf7673d WatchSource:0}: Error finding container caa459baea479464f64201d130df9e5c838bc0c2fb3129cd3edbfb5d8cf7673d: Status 404 returned error can't find the container with id caa459baea479464f64201d130df9e5c838bc0c2fb3129cd3edbfb5d8cf7673d
Jul 06 16:08:21.543223 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.543200    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:21.984205 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.984161    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" event={"ID":"249d4ff4-c4e4-4244-aedd-15d378b98786","Type":"ContainerStarted","Data":"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"}
Jul 06 16:08:21.984205 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:21.984212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" event={"ID":"249d4ff4-c4e4-4244-aedd-15d378b98786","Type":"ContainerStarted","Data":"caa459baea479464f64201d130df9e5c838bc0c2fb3129cd3edbfb5d8cf7673d"}
Jul 06 16:08:22.007890 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:22.007843    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" podStartSLOduration=1.007826371 podStartE2EDuration="1.007826371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:08:22.006947045 +0000 UTC m=+2068.876994212" watchObservedRunningTime="2026-07-06 16:08:22.007826371 +0000 UTC m=+2068.877873539"
Jul 06 16:08:31.385045 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:31.385000    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:31.385476 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:31.385058    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:31.397493 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:31.397467    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:32.046543 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:32.046511    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:39.342898 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:39.342853    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:40.336500 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.336466    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:08:40.336757 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.336728    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" podUID="8237656f-3a4a-4db7-80fa-fa78b66296cc" containerName="maas-api" containerID="cri-o://469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba" gracePeriod=30
Jul 06 16:08:40.594067 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.594007    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:08:40.695263 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.695220    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjctx\" (UniqueName: \"kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx\") pod \"8237656f-3a4a-4db7-80fa-fa78b66296cc\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") "
Jul 06 16:08:40.695466 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.695327    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") pod \"8237656f-3a4a-4db7-80fa-fa78b66296cc\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") "
Jul 06 16:08:40.695466 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.695365    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca\") pod \"8237656f-3a4a-4db7-80fa-fa78b66296cc\" (UID: \"8237656f-3a4a-4db7-80fa-fa78b66296cc\") "
Jul 06 16:08:40.695883 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.695787    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8237656f-3a4a-4db7-80fa-fa78b66296cc" (UID: "8237656f-3a4a-4db7-80fa-fa78b66296cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:08:40.697514 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.697486    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx" (OuterVolumeSpecName: "kube-api-access-vjctx") pod "8237656f-3a4a-4db7-80fa-fa78b66296cc" (UID: "8237656f-3a4a-4db7-80fa-fa78b66296cc"). InnerVolumeSpecName "kube-api-access-vjctx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:08:40.697628 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.697488    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8237656f-3a4a-4db7-80fa-fa78b66296cc" (UID: "8237656f-3a4a-4db7-80fa-fa78b66296cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:08:40.796493 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.796455    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjctx\" (UniqueName: \"kubernetes.io/projected/8237656f-3a4a-4db7-80fa-fa78b66296cc-kube-api-access-vjctx\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:40.796493 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.796486    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8237656f-3a4a-4db7-80fa-fa78b66296cc-maas-api-tls\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:40.796493 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:40.796495    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237656f-3a4a-4db7-80fa-fa78b66296cc-openshift-service-ca\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:41.079263 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.079224    2567 generic.go:356] "Generic (PLEG): container finished" podID="8237656f-3a4a-4db7-80fa-fa78b66296cc" containerID="469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba" exitCode=0
Jul 06 16:08:41.079443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.079292    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"
Jul 06 16:08:41.079443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.079301    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" event={"ID":"8237656f-3a4a-4db7-80fa-fa78b66296cc","Type":"ContainerDied","Data":"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"}
Jul 06 16:08:41.079443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.079402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t" event={"ID":"8237656f-3a4a-4db7-80fa-fa78b66296cc","Type":"ContainerDied","Data":"e40fac1c4882e01ee469dc687ebf59b9acbbc59f50d805ea416f0dc2d7574833"}
Jul 06 16:08:41.079443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.079432    2567 scope.go:117] "RemoveContainer" containerID="469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"
Jul 06 16:08:41.088684 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.088646    2567 scope.go:117] "RemoveContainer" containerID="469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"
Jul 06 16:08:41.089028 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:08:41.089008    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba\": container with ID starting with 469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba not found: ID does not exist" containerID="469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"
Jul 06 16:08:41.089079 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.089037    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba"} err="failed to get container status \"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba\": rpc error: code = NotFound desc = could not find container \"469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba\": container with ID starting with 469e8a776011fcc9a4b7deeaae1521fe1e554c8afdea530706c27a5740d78dba not found: ID does not exist"
Jul 06 16:08:41.109469 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.109434    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:08:41.112382 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.112357    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-afeb72-b7f6bfdcf-72x5t"]
Jul 06 16:08:41.664825 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:41.664794    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8237656f-3a4a-4db7-80fa-fa78b66296cc" path="/var/lib/kubelet/pods/8237656f-3a4a-4db7-80fa-fa78b66296cc/volumes"
Jul 06 16:08:54.364157 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.364119    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" podUID="249d4ff4-c4e4-4244-aedd-15d378b98786" containerName="main" containerID="cri-o://e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835" gracePeriod=45
Jul 06 16:08:54.619866 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.619805    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:54.722422 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722389    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722435    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-786nr\" (UniqueName: \"kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722586 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722544    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722738 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722588    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722738 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722639    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home\") pod \"249d4ff4-c4e4-4244-aedd-15d378b98786\" (UID: \"249d4ff4-c4e4-4244-aedd-15d378b98786\") "
Jul 06 16:08:54.722738 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722696    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:08:54.722892 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722836    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache" (OuterVolumeSpecName: "model-cache") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:08:54.723013 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.722985    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home" (OuterVolumeSpecName: "home") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:08:54.723013 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.723005    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:54.723179 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.723024    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:54.724924 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.724898    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm" (OuterVolumeSpecName: "dshm") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:08:54.725020 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.724973    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:08:54.725020 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.724976    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr" (OuterVolumeSpecName: "kube-api-access-786nr") pod "249d4ff4-c4e4-4244-aedd-15d378b98786" (UID: "249d4ff4-c4e4-4244-aedd-15d378b98786"). InnerVolumeSpecName "kube-api-access-786nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:08:54.824093 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.824063    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249d4ff4-c4e4-4244-aedd-15d378b98786-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:54.824093 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.824090    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:54.824093 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.824099    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249d4ff4-c4e4-4244-aedd-15d378b98786-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:54.824310 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:54.824107    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786nr\" (UniqueName: \"kubernetes.io/projected/249d4ff4-c4e4-4244-aedd-15d378b98786-kube-api-access-786nr\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:08:55.148186 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.148152    2567 generic.go:356] "Generic (PLEG): container finished" podID="249d4ff4-c4e4-4244-aedd-15d378b98786" containerID="e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835" exitCode=0
Jul 06 16:08:55.148376 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.148261    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"
Jul 06 16:08:55.148376 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.148276    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" event={"ID":"249d4ff4-c4e4-4244-aedd-15d378b98786","Type":"ContainerDied","Data":"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"}
Jul 06 16:08:55.148376 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.148319    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85" event={"ID":"249d4ff4-c4e4-4244-aedd-15d378b98786","Type":"ContainerDied","Data":"caa459baea479464f64201d130df9e5c838bc0c2fb3129cd3edbfb5d8cf7673d"}
Jul 06 16:08:55.148376 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.148340    2567 scope.go:117] "RemoveContainer" containerID="e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"
Jul 06 16:08:55.158482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.158449    2567 scope.go:117] "RemoveContainer" containerID="e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"
Jul 06 16:08:55.158826 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:08:55.158800    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835\": container with ID starting with e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835 not found: ID does not exist" containerID="e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"
Jul 06 16:08:55.158965 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.158834    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835"} err="failed to get container status \"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835\": rpc error: code = NotFound desc = could not find container \"e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835\": container with ID starting with e864d9a13d5d6d395c8fdda1cfaa6797f4027b9174e558dd7ed3a35ee4e19835 not found: ID does not exist"
Jul 06 16:08:55.172031 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.172000    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:55.176108 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.176082    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-785a99/test-model-785a99-kserve-5c99496cc9-fxl85"]
Jul 06 16:08:55.664463 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:08:55.664422    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249d4ff4-c4e4-4244-aedd-15d378b98786" path="/var/lib/kubelet/pods/249d4ff4-c4e4-4244-aedd-15d378b98786/volumes"
Jul 06 16:09:22.822657 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:22.822621    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:09:22.823201 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:22.823109    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" containerID="cri-o://20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded" gracePeriod=30
Jul 06 16:09:22.868779 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:22.868736    2567 patch_prober.go:28] interesting pod/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 16:09:22.868941 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:22.868801    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:09:28.083043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.082973    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:09:28.207356 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207316    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfjf6\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207356 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207364    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207588 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207528    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207588 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207571    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207726 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207596    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207726 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207627    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207726 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207695    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207888 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207727    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207888 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207753    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert\") pod \"0ab80143-a705-4cb5-bb4e-a2e0add08857\" (UID: \"0ab80143-a705-4cb5-bb4e-a2e0add08857\") "
Jul 06 16:09:28.207888 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207847    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:09:28.208030 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.207923    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:09:28.208093 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208034    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:09:28.208177 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208151    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:09:28.208293 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208271    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.208357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208302    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.208357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208317    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.208357 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208330    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab80143-a705-4cb5-bb4e-a2e0add08857-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.208707 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.208558    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:09:28.209867 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.209844    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:09:28.209867 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.209843    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6" (OuterVolumeSpecName: "kube-api-access-mfjf6") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "kube-api-access-mfjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:09:28.210044 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.210028    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:09:28.210151 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.210131    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ab80143-a705-4cb5-bb4e-a2e0add08857" (UID: "0ab80143-a705-4cb5-bb4e-a2e0add08857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:09:28.309208 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.309164    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.309208 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.309198    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.309208 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.309207    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfjf6\" (UniqueName: \"kubernetes.io/projected/0ab80143-a705-4cb5-bb4e-a2e0add08857-kube-api-access-mfjf6\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.309208 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.309216    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.309483 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.309224    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab80143-a705-4cb5-bb4e-a2e0add08857-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:09:28.317687 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.317634    2567 generic.go:356] "Generic (PLEG): container finished" podID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerID="20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded" exitCode=0
Jul 06 16:09:28.317846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.317705    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerDied","Data":"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"}
Jul 06 16:09:28.317846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.317714    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"
Jul 06 16:09:28.317846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.317750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr" event={"ID":"0ab80143-a705-4cb5-bb4e-a2e0add08857","Type":"ContainerDied","Data":"2db79c4b92b3595f568a64e24f799a014e8fab7c53132a1a696a394836c81dad"}
Jul 06 16:09:28.317846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.317774    2567 scope.go:117] "RemoveContainer" containerID="20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"
Jul 06 16:09:28.327051 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.327025    2567 scope.go:117] "RemoveContainer" containerID="d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"
Jul 06 16:09:28.335422 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.335403    2567 scope.go:117] "RemoveContainer" containerID="20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"
Jul 06 16:09:28.335731 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:09:28.335705    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded\": container with ID starting with 20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded not found: ID does not exist" containerID="20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"
Jul 06 16:09:28.335797 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.335741    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded"} err="failed to get container status \"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded\": rpc error: code = NotFound desc = could not find container \"20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded\": container with ID starting with 20293572a465934538efdd333d6110d085fc504b2b19ba536820df54c1337ded not found: ID does not exist"
Jul 06 16:09:28.335797 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.335759    2567 scope.go:117] "RemoveContainer" containerID="d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"
Jul 06 16:09:28.336002 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:09:28.335985    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da\": container with ID starting with d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da not found: ID does not exist" containerID="d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"
Jul 06 16:09:28.336043 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.336008    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da"} err="failed to get container status \"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da\": rpc error: code = NotFound desc = could not find container \"d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da\": container with ID starting with d747a8946a77ed652731b238032a563ea97ab4cc7e94aab92ca2ab5eeeb618da not found: ID does not exist"
Jul 06 16:09:28.340319 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.340296    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:09:28.344314 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:28.344292    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-afeb72-openshift-default-7c6dff66f5-q2rbr"]
Jul 06 16:09:29.665651 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:29.665617    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" path="/var/lib/kubelet/pods/0ab80143-a705-4cb5-bb4e-a2e0add08857/volumes"
Jul 06 16:09:31.076889 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.076846    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:09:31.077467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077442    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:09:31.077656 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077639    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:09:31.077809 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077784    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8237656f-3a4a-4db7-80fa-fa78b66296cc" containerName="maas-api"
Jul 06 16:09:31.077917 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077813    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8237656f-3a4a-4db7-80fa-fa78b66296cc" containerName="maas-api"
Jul 06 16:09:31.077917 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077858    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:09:31.077917 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077868    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:09:31.077917 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077909    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249d4ff4-c4e4-4244-aedd-15d378b98786" containerName="main"
Jul 06 16:09:31.077917 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.077918    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="249d4ff4-c4e4-4244-aedd-15d378b98786" containerName="main"
Jul 06 16:09:31.078142 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.078030    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:09:31.078142 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.078042    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="249d4ff4-c4e4-4244-aedd-15d378b98786" containerName="main"
Jul 06 16:09:31.078142 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.078055    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8237656f-3a4a-4db7-80fa-fa78b66296cc" containerName="maas-api"
Jul 06 16:09:31.082764 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.082746    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.085239 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.085220    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-34ada1-openshift-default-dockercfg-vqbxg\""
Jul 06 16:09:31.094996 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.094975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:09:31.137590 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.137792 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qd9\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.137792 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.137792 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.137792 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.138019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137827    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.138019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.138019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.138019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.137942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238678 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.238852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.238845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239052    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239111    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239358 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qd9\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239414 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239365    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239474 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239420    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.239474 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.239465    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.240998 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.240971    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.241321 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.241303    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.247655 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.247632    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qd9\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.247765 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.247635    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token\") pod \"e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") " pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.393174 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.393085    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:31.539519 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.539494    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:09:31.541836 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:09:31.541805    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffb9411_bf48_4e5d_aad2_2642a9c9c19c.slice/crio-20bc67ed0ed299d6ecea44872495c029e97b60f0724717ebfd1b64c5c4081b6c WatchSource:0}: Error finding container 20bc67ed0ed299d6ecea44872495c029e97b60f0724717ebfd1b64c5c4081b6c: Status 404 returned error can't find the container with id 20bc67ed0ed299d6ecea44872495c029e97b60f0724717ebfd1b64c5c4081b6c
Jul 06 16:09:31.544059 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.544013    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:31.544140 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.544121    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:31.544186 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:31.544169    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:32.341741 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:32.341699    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerStarted","Data":"9090863f20cbacc0dea8cbeb5510fa0d12cd947d926ea81c2d882095e8bbcac9"}
Jul 06 16:09:32.342199 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:32.341766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerStarted","Data":"20bc67ed0ed299d6ecea44872495c029e97b60f0724717ebfd1b64c5c4081b6c"}
Jul 06 16:09:32.369759 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:32.369711    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podStartSLOduration=1.369694533 podStartE2EDuration="1.369694533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:09:32.36859523 +0000 UTC m=+2139.238642398" watchObservedRunningTime="2026-07-06 16:09:32.369694533 +0000 UTC m=+2139.239741700"
Jul 06 16:09:32.393452 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:32.393420    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:32.398507 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:32.398484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:33.346727 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:33.346696    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:33.347588 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:33.347568    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:48.348677 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.348632    2567 patch_prober.go:28] interesting pod/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 06 16:09:48.349103 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.348742    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 06 16:09:48.437921 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.437887    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerID="9090863f20cbacc0dea8cbeb5510fa0d12cd947d926ea81c2d882095e8bbcac9" exitCode=0
Jul 06 16:09:48.438114 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.437962    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerDied","Data":"9090863f20cbacc0dea8cbeb5510fa0d12cd947d926ea81c2d882095e8bbcac9"}
Jul 06 16:09:48.438375 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.438360    2567 scope.go:117] "RemoveContainer" containerID="9090863f20cbacc0dea8cbeb5510fa0d12cd947d926ea81c2d882095e8bbcac9"
Jul 06 16:09:48.439339 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.439312    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:48.439398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.439379    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:48.439435 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:48.439415    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:09:49.445398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:49.445363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerStarted","Data":"6f37acba9fac82dba67aa89ac790804bcc8be63f9e8ff2ecf49ff430875366aa"}
Jul 06 16:09:50.394030 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:50.393994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:50.394432 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:50.394340    2567 patch_prober.go:28] interesting pod/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 06 16:09:50.394600 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:50.394578    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 06 16:09:51.394368 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:51.394328    2567 patch_prober.go:28] interesting pod/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 06 16:09:51.395006 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:51.394974    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 06 16:09:52.394894 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:52.394389    2567 patch_prober.go:28] interesting pod/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 06 16:09:52.394894 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:52.394458    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 06 16:09:53.394080 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:53.394041    2567 patch_prober.go:28] interesting pod/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 06 16:09:53.394289 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:53.394103    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 06 16:09:54.398339 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:54.398309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:54.398875 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:54.398582    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:09:54.399218 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:09:54.399202    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:10:03.329482 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:03.329447    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:10:03.329960 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:03.329770    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy" containerID="cri-o://6f37acba9fac82dba67aa89ac790804bcc8be63f9e8ff2ecf49ff430875366aa" gracePeriod=30
Jul 06 16:10:08.543013 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.542976    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerID="6f37acba9fac82dba67aa89ac790804bcc8be63f9e8ff2ecf49ff430875366aa" exitCode=0
Jul 06 16:10:08.543490 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.543040    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerDied","Data":"6f37acba9fac82dba67aa89ac790804bcc8be63f9e8ff2ecf49ff430875366aa"}
Jul 06 16:10:08.543490 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.543098    2567 scope.go:117] "RemoveContainer" containerID="9090863f20cbacc0dea8cbeb5510fa0d12cd947d926ea81c2d882095e8bbcac9"
Jul 06 16:10:08.595966 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.595910    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:10:08.679130 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679097    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qd9\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679140    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679171    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679203    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679224    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679322 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679244    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679587 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679587 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679366    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679587 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679391    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket\") pod \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\" (UID: \"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c\") "
Jul 06 16:10:08.679587 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679528    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:10:08.679846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679593    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:10:08.679846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679725    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.679846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679743    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.679846 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.679595    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:10:08.680137 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.680114    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:10:08.680262 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.680185    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:10:08.681870 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.681847    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:10:08.681953 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.681900    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:10:08.682101 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.682086    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9" (OuterVolumeSpecName: "kube-api-access-75qd9") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "kube-api-access-75qd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:10:08.682158 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.682101    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" (UID: "9ffb9411-bf48-4e5d-aad2-2642a9c9c19c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:10:08.780338 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780306    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780338 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780333    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780338 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780342    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780590 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780350    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780590 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780359    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780590 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780367    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qd9\" (UniqueName: \"kubernetes.io/projected/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-kube-api-access-75qd9\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:08.780590 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:08.780376    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:10:09.551503 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.551468    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"
Jul 06 16:10:09.551982 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.551478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks" event={"ID":"9ffb9411-bf48-4e5d-aad2-2642a9c9c19c","Type":"ContainerDied","Data":"20bc67ed0ed299d6ecea44872495c029e97b60f0724717ebfd1b64c5c4081b6c"}
Jul 06 16:10:09.551982 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.551604    2567 scope.go:117] "RemoveContainer" containerID="6f37acba9fac82dba67aa89ac790804bcc8be63f9e8ff2ecf49ff430875366aa"
Jul 06 16:10:09.593195 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.593162    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:10:09.605313 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.605283    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-34ada1-openshift-default-7d85b899d7-cb5ks"]
Jul 06 16:10:09.664980 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:09.664948    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" path="/var/lib/kubelet/pods/9ffb9411-bf48-4e5d-aad2-2642a9c9c19c/volumes"
Jul 06 16:10:12.415398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415361    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415759    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415774    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415793    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415801    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415895    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:12.415939 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.415907    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab80143-a705-4cb5-bb4e-a2e0add08857" containerName="istio-proxy"
Jul 06 16:10:12.419101 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.419080    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.428608 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.428586    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c8a9d1-openshift-default-dockercfg-zvpj8\""
Jul 06 16:10:12.462642 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.462610    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:10:12.512719 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.512927 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.512927 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.512927 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.512927 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.512927 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.513252 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkwrs\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.513252 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.512985    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.513252 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.513016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613461 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613461 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkwrs\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.613769 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614026 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614026 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614026 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613918    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614026 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.613931    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614026 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.614022    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614231 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.614197    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.614411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.614384    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.616011 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.615990    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.616306 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.616289    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.636852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.636827    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.636987 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.636910    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkwrs\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs\") pod \"e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") " pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.729528 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.729489    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:12.897870 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.897843    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:10:12.899595 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:10:12.899568    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8572acad_677b_40a5_a0d6_ab42d31d8ea2.slice/crio-28422459b00b264803080e7de3be37d8b7e2d02e0a29abf26b8514011d68f503 WatchSource:0}: Error finding container 28422459b00b264803080e7de3be37d8b7e2d02e0a29abf26b8514011d68f503: Status 404 returned error can't find the container with id 28422459b00b264803080e7de3be37d8b7e2d02e0a29abf26b8514011d68f503
Jul 06 16:10:12.902049 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.902015    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:12.902113 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.902083    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:12.902146 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:12.902110    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:13.576013 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:13.575971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerStarted","Data":"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"}
Jul 06 16:10:13.576501 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:13.576038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerStarted","Data":"28422459b00b264803080e7de3be37d8b7e2d02e0a29abf26b8514011d68f503"}
Jul 06 16:10:13.605658 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:13.605604    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podStartSLOduration=1.605588069 podStartE2EDuration="1.605588069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:10:13.601614693 +0000 UTC m=+2180.471661860" watchObservedRunningTime="2026-07-06 16:10:13.605588069 +0000 UTC m=+2180.475635662"
Jul 06 16:10:13.730032 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:13.729995    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:13.735192 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:13.735171    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:14.581560 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:14.581520    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:14.582483 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:14.582463    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:20.994699 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:20.991330    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:10:21.001693 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.000140    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffb9411-bf48-4e5d-aad2-2642a9c9c19c" containerName="istio-proxy"
Jul 06 16:10:21.006804 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.006777    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.009387 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.009364    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b747bf-openshift-default-dockercfg-mmbsg\""
Jul 06 16:10:21.013185 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.013158    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:10:21.094215 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.094173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jdh\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.094531 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.094509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.094717 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.094699    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.094887 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.094872    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.095038 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.095022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.095202 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.095187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.095371 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.095356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.095530 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.095515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.095718 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.095701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.196710 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.196633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.196898 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.196719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.196898 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.196792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.197027 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.196892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.197206 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.197184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.197287 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.197229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.197416 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.197395    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.197903 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.197861    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.198192 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.198128    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.198436 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.197309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.198616 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.198596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jdh\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.198759 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.198739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.199055 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.198983    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.199276 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.199230    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.200196 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.200165    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.200313 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.200296    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.209260 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.209237    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.214575 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.214550    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jdh\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh\") pod \"e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") " pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.320358 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.320263    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:21.498706 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.488347    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:10:21.501096 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:10:21.501059    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a71d6b7_833c_43fc_9a62_336a0935659d.slice/crio-0cb349acbc82d7146d1be69008c8c0d09ac47d18755afca4a809370d0d6bdb2a WatchSource:0}: Error finding container 0cb349acbc82d7146d1be69008c8c0d09ac47d18755afca4a809370d0d6bdb2a: Status 404 returned error can't find the container with id 0cb349acbc82d7146d1be69008c8c0d09ac47d18755afca4a809370d0d6bdb2a
Jul 06 16:10:21.503062 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.503029    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:21.503133 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.503104    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:21.503175 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.503134    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:21.616538 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.616455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerStarted","Data":"10b6aafb8f3584fba46bf56cc0bc83c045ba9225614a4efd28420eae20773c3d"}
Jul 06 16:10:21.616538 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.616512    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerStarted","Data":"0cb349acbc82d7146d1be69008c8c0d09ac47d18755afca4a809370d0d6bdb2a"}
Jul 06 16:10:21.647293 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:21.647226    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podStartSLOduration=1.647205126 podStartE2EDuration="1.647205126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:10:21.647205658 +0000 UTC m=+2188.517252827" watchObservedRunningTime="2026-07-06 16:10:21.647205126 +0000 UTC m=+2188.517252294"
Jul 06 16:10:22.321430 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:22.321391    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:22.326388 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:22.326362    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:22.622855 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:22.622774    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:22.623852 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:22.623835    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:27.649626 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.649595    2567 generic.go:356] "Generic (PLEG): container finished" podID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerID="8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5" exitCode=0
Jul 06 16:10:27.650056 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.649681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerDied","Data":"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"}
Jul 06 16:10:27.650102 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.650086    2567 scope.go:117] "RemoveContainer" containerID="8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"
Jul 06 16:10:27.651130 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.651100    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:27.651243 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.651187    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:27.651243 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:27.651231    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:28.656450 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:28.656415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerStarted","Data":"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"}
Jul 06 16:10:28.730381 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:28.730348    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:28.731224 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:28.731188    2567 patch_prober.go:28] interesting pod/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 06 16:10:28.731275 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:28.731247    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 06 16:10:29.583073 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:29.583010    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:29.729942 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:29.729903    2567 patch_prober.go:28] interesting pod/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 06 16:10:29.730332 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:29.729967    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 06 16:10:30.730136 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:30.730100    2567 patch_prober.go:28] interesting pod/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 06 16:10:30.730535 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:30.730160    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 06 16:10:31.730428 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:31.730340    2567 patch_prober.go:28] interesting pod/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 06 16:10:31.730428 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:31.730398    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 06 16:10:32.735943 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:32.735913    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:32.737075 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:32.737053    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:10:34.686092 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.685986    2567 generic.go:356] "Generic (PLEG): container finished" podID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerID="10b6aafb8f3584fba46bf56cc0bc83c045ba9225614a4efd28420eae20773c3d" exitCode=0
Jul 06 16:10:34.686092 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.686058    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerDied","Data":"10b6aafb8f3584fba46bf56cc0bc83c045ba9225614a4efd28420eae20773c3d"}
Jul 06 16:10:34.686524 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.686438    2567 scope.go:117] "RemoveContainer" containerID="10b6aafb8f3584fba46bf56cc0bc83c045ba9225614a4efd28420eae20773c3d"
Jul 06 16:10:34.687554 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.687515    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:34.687628 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.687607    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:34.687685 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:34.687653    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 06 16:10:35.696924 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:35.696882    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerStarted","Data":"56849435299b68df94e8adaa9f06d1cb9fcc03e7ea77f3266287ad69317970ca"}
Jul 06 16:10:36.045882 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.045795    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:10:36.048636 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.048612    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.051049 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.051021    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"openshift-service-ca.crt\""
Jul 06 16:10:36.051177 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.051132    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"auth-test-model-c8a9d1-kserve-self-signed-certs\""
Jul 06 16:10:36.051774 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.051759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"kube-root-ca.crt\""
Jul 06 16:10:36.051873 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.051761    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"default-dockercfg-fkgd7\""
Jul 06 16:10:36.062893 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.062872    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:10:36.156577 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.156835 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.156835 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.156835 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.157015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156889    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.157015 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.156923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5h8m\" (UniqueName: \"kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257508 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257727 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257727 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257727 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257914 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5h8m\" (UniqueName: \"kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257914 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.257914 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257884    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.258037 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257939    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.258037 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.257981    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.260048 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.260016    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.260349 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.260331    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.266845 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.266821    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5h8m\" (UniqueName: \"kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m\") pod \"auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") " pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.321163 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.321070    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:36.322345 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.322319    2567 patch_prober.go:28] interesting pod/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 16:10:36.322463 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.322391    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 16:10:36.359681 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.359644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:36.543772 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.543744    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:10:36.544622 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:10:36.544598    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9a16f75_a8e0_45eb_b423_91e86586b325.slice/crio-25e49512236bbec7926b35e14023e17735ff32b24deffe329bd29fc2ca640f70 WatchSource:0}: Error finding container 25e49512236bbec7926b35e14023e17735ff32b24deffe329bd29fc2ca640f70: Status 404 returned error can't find the container with id 25e49512236bbec7926b35e14023e17735ff32b24deffe329bd29fc2ca640f70
Jul 06 16:10:36.704095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.704062    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" event={"ID":"a9a16f75-a8e0-45eb-b423-91e86586b325","Type":"ContainerStarted","Data":"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"}
Jul 06 16:10:36.704465 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.704106    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" event={"ID":"a9a16f75-a8e0-45eb-b423-91e86586b325","Type":"ContainerStarted","Data":"25e49512236bbec7926b35e14023e17735ff32b24deffe329bd29fc2ca640f70"}
Jul 06 16:10:36.724324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:36.724276    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" podStartSLOduration=0.72426164 podStartE2EDuration="724.26164ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:10:36.723402734 +0000 UTC m=+2203.593449899" watchObservedRunningTime="2026-07-06 16:10:36.72426164 +0000 UTC m=+2203.594308809"
Jul 06 16:10:37.320902 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:37.320868    2567 patch_prober.go:28] interesting pod/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 16:10:37.321073 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:37.320925    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 16:10:37.622785 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:37.622699    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:38.321594 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:38.321559    2567 patch_prober.go:28] interesting pod/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 16:10:38.322021 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:38.321615    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 16:10:39.321067 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:39.321024    2567 patch_prober.go:28] interesting pod/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 16:10:39.321259 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:39.321089    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 16:10:40.325328 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:40.325291    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:40.326610 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:40.326588    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:10:46.360364 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:46.360319    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:46.360818 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:46.360378    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:46.373295 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:46.373265    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:10:46.763867 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:10:46.763838    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:13:35.665100 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:35.665069    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:13:50.685887 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:50.685840    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" podUID="a9a16f75-a8e0-45eb-b423-91e86586b325" containerName="main" containerID="cri-o://cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c" gracePeriod=45
Jul 06 16:13:50.945246 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:50.945216    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:13:51.024129 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024145    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024175    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024214    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024266    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5h8m\" (UniqueName: \"kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024324 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024306    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache\") pod \"a9a16f75-a8e0-45eb-b423-91e86586b325\" (UID: \"a9a16f75-a8e0-45eb-b423-91e86586b325\") "
Jul 06 16:13:51.024595 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024374    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:13:51.024595 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024535    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home" (OuterVolumeSpecName: "home") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:13:51.024733 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024630    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.024733 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024647    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.024733 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.024644    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:13:51.026614 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.026591    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm" (OuterVolumeSpecName: "dshm") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:13:51.026715 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.026651    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:13:51.026715 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.026690    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m" (OuterVolumeSpecName: "kube-api-access-x5h8m") pod "a9a16f75-a8e0-45eb-b423-91e86586b325" (UID: "a9a16f75-a8e0-45eb-b423-91e86586b325"). InnerVolumeSpecName "kube-api-access-x5h8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:13:51.125081 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.125052    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.125081 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.125077    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a16f75-a8e0-45eb-b423-91e86586b325-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.125081 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.125087    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5h8m\" (UniqueName: \"kubernetes.io/projected/a9a16f75-a8e0-45eb-b423-91e86586b325-kube-api-access-x5h8m\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.125298 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.125095    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a16f75-a8e0-45eb-b423-91e86586b325-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:13:51.650799 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.650758    2567 generic.go:356] "Generic (PLEG): container finished" podID="a9a16f75-a8e0-45eb-b423-91e86586b325" containerID="cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c" exitCode=0
Jul 06 16:13:51.650979 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.650811    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"
Jul 06 16:13:51.650979 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.650850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" event={"ID":"a9a16f75-a8e0-45eb-b423-91e86586b325","Type":"ContainerDied","Data":"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"}
Jul 06 16:13:51.650979 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.650909    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8" event={"ID":"a9a16f75-a8e0-45eb-b423-91e86586b325","Type":"ContainerDied","Data":"25e49512236bbec7926b35e14023e17735ff32b24deffe329bd29fc2ca640f70"}
Jul 06 16:13:51.650979 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.650936    2567 scope.go:117] "RemoveContainer" containerID="cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"
Jul 06 16:13:51.661879 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.661859    2567 scope.go:117] "RemoveContainer" containerID="cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"
Jul 06 16:13:51.662154 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:13:51.662129    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c\": container with ID starting with cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c not found: ID does not exist" containerID="cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"
Jul 06 16:13:51.662216 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.662167    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c"} err="failed to get container status \"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c\": rpc error: code = NotFound desc = could not find container \"cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c\": container with ID starting with cbb22dd39856ba2f8c64a7974e95ac0850312b685849caf124c7fc2ec099b80c not found: ID does not exist"
Jul 06 16:13:51.674082 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.674054    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:13:51.678115 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:51.678093    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/auth-test-model-c8a9d1-kserve-6894bd8dcf-fjzd8"]
Jul 06 16:13:53.670728 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:13:53.670693    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9a16f75-a8e0-45eb-b423-91e86586b325" path="/var/lib/kubelet/pods/a9a16f75-a8e0-45eb-b423-91e86586b325/volumes"
Jul 06 16:16:27.836996 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.836963    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:16:27.837602 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.837585    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a16f75-a8e0-45eb-b423-91e86586b325" containerName="main"
Jul 06 16:16:27.837703 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.837606    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a16f75-a8e0-45eb-b423-91e86586b325" containerName="main"
Jul 06 16:16:27.837758 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.837713    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a16f75-a8e0-45eb-b423-91e86586b325" containerName="main"
Jul 06 16:16:27.839790 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.839775    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.842058 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.842037    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"openshift-service-ca.crt\""
Jul 06 16:16:27.842896 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.842877    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"default-dockercfg-qw6cg\""
Jul 06 16:16:27.842969 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.842916    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"kube-root-ca.crt\""
Jul 06 16:16:27.843007 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.842881    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"rate-test-model-b747bf-kserve-self-signed-certs\""
Jul 06 16:16:27.853462 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.853439    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:16:27.875417 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.875556 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.875556 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.875556 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.875722 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875604    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lb67\" (UniqueName: \"kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.875722 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.875631    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.976771 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.976771 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lb67\" (UniqueName: \"kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977008 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.976945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977209 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.977185    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977209 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.977202    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.977346 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.977324    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.979271 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.979242    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.979514 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.979498    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:27.987549 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:27.987528    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lb67\" (UniqueName: \"kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67\") pod \"rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") " pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:28.148901 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:28.148799    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:28.284001 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:28.283973    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:16:28.285774 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:16:28.285748    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b63c451_c60f_4308_9650_a59d5c0dc023.slice/crio-40255e6e8be29eaf31667689322c985e63ae93f52eae0461cd5636101d1b7e56 WatchSource:0}: Error finding container 40255e6e8be29eaf31667689322c985e63ae93f52eae0461cd5636101d1b7e56: Status 404 returned error can't find the container with id 40255e6e8be29eaf31667689322c985e63ae93f52eae0461cd5636101d1b7e56
Jul 06 16:16:28.411087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:28.410983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" event={"ID":"9b63c451-c60f-4308-9650-a59d5c0dc023","Type":"ContainerStarted","Data":"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"}
Jul 06 16:16:28.411087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:28.411035    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" event={"ID":"9b63c451-c60f-4308-9650-a59d5c0dc023","Type":"ContainerStarted","Data":"40255e6e8be29eaf31667689322c985e63ae93f52eae0461cd5636101d1b7e56"}
Jul 06 16:16:28.429567 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:28.429515    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" podStartSLOduration=1.429500516 podStartE2EDuration="1.429500516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:16:28.426883424 +0000 UTC m=+2555.296930592" watchObservedRunningTime="2026-07-06 16:16:28.429500516 +0000 UTC m=+2555.299547702"
Jul 06 16:16:38.149926 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:38.149885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:38.150423 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:38.150179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:38.162796 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:38.162767    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:16:38.468892 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:16:38.468864    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:17:12.972905 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:12.972861    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:17:27.993598 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:27.993483    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" podUID="9b63c451-c60f-4308-9650-a59d5c0dc023" containerName="main" containerID="cri-o://e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43" gracePeriod=45
Jul 06 16:17:28.245381 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.245320    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:17:28.362242 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362203    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362441 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362261    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362441 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362297    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lb67\" (UniqueName: \"kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362441 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362364    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362441 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362417    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362659 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362445    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs\") pod \"9b63c451-c60f-4308-9650-a59d5c0dc023\" (UID: \"9b63c451-c60f-4308-9650-a59d5c0dc023\") "
Jul 06 16:17:28.362659 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362531    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:17:28.362659 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362603    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home" (OuterVolumeSpecName: "home") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:17:28.362821 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362705    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:17:28.362871 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362833    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-home\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.362871 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362846    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-tmp-dir\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.362871 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.362855    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-model-cache\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.364827 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.364797    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 16:17:28.364827 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.364806    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm" (OuterVolumeSpecName: "dshm") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:17:28.364827 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.364819    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67" (OuterVolumeSpecName: "kube-api-access-4lb67") pod "9b63c451-c60f-4308-9650-a59d5c0dc023" (UID: "9b63c451-c60f-4308-9650-a59d5c0dc023"). InnerVolumeSpecName "kube-api-access-4lb67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:17:28.464153 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.464123    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b63c451-c60f-4308-9650-a59d5c0dc023-dshm\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.464153 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.464150    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b63c451-c60f-4308-9650-a59d5c0dc023-tls-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.464153 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.464160    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lb67\" (UniqueName: \"kubernetes.io/projected/9b63c451-c60f-4308-9650-a59d5c0dc023-kube-api-access-4lb67\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:17:28.704053 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.704021    2567 generic.go:356] "Generic (PLEG): container finished" podID="9b63c451-c60f-4308-9650-a59d5c0dc023" containerID="e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43" exitCode=0
Jul 06 16:17:28.704220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.704073    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"
Jul 06 16:17:28.704220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.704090    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" event={"ID":"9b63c451-c60f-4308-9650-a59d5c0dc023","Type":"ContainerDied","Data":"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"}
Jul 06 16:17:28.704220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.704124    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr" event={"ID":"9b63c451-c60f-4308-9650-a59d5c0dc023","Type":"ContainerDied","Data":"40255e6e8be29eaf31667689322c985e63ae93f52eae0461cd5636101d1b7e56"}
Jul 06 16:17:28.704220 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.704141    2567 scope.go:117] "RemoveContainer" containerID="e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"
Jul 06 16:17:28.713488 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.713468    2567 scope.go:117] "RemoveContainer" containerID="e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"
Jul 06 16:17:28.713741 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:17:28.713720    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43\": container with ID starting with e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43 not found: ID does not exist" containerID="e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"
Jul 06 16:17:28.713811 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.713750    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43"} err="failed to get container status \"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43\": rpc error: code = NotFound desc = could not find container \"e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43\": container with ID starting with e56cfd2388c375a9ef611808e576f7e96426af0d3706732b533892a3def1aa43 not found: ID does not exist"
Jul 06 16:17:28.726235 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.726208    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:17:28.731977 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:28.731956    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/rate-test-model-b747bf-kserve-c5bbb4c5f-d6sbr"]
Jul 06 16:17:29.665351 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:29.665318    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b63c451-c60f-4308-9650-a59d5c0dc023" path="/var/lib/kubelet/pods/9b63c451-c60f-4308-9650-a59d5c0dc023/volumes"
Jul 06 16:17:58.041300 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:58.041267    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:17:58.041786 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:17:58.041484    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" containerID="cri-o://858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837" gracePeriod=30
Jul 06 16:18:02.738981 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:02.738938    2567 patch_prober.go:28] interesting pod/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 16:18:02.739382 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:02.739002    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:18:03.299344 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.299318    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:18:03.387184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387133    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387184 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387176    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkwrs\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387232    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387310    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387443 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387418    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387645 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387451    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:03.387645 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387456    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387645 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387505    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs\") pod \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\" (UID: \"8572acad-677b-40a5-a0d6-ab42d31d8ea2\") "
Jul 06 16:18:03.387878 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387839    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:03.387878 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387863    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.388082 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.387875    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:18:03.388082 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.388039    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:03.388623 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.388594    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data" (OuterVolumeSpecName: "istio-data") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:03.389729 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.389703    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token" (OuterVolumeSpecName: "istio-token") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:18:03.389817 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.389726    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs" (OuterVolumeSpecName: "kube-api-access-gkwrs") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "kube-api-access-gkwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:18:03.389817 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.389807    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:18:03.389943 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.389925    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8572acad-677b-40a5-a0d6-ab42d31d8ea2" (UID: "8572acad-677b-40a5-a0d6-ab42d31d8ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:03.489171 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489137    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489171 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489166    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489171 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489176    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489185    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489193    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8572acad-677b-40a5-a0d6-ab42d31d8ea2-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489200    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489208    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkwrs\" (UniqueName: \"kubernetes.io/projected/8572acad-677b-40a5-a0d6-ab42d31d8ea2-kube-api-access-gkwrs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.489467 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.489215    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8572acad-677b-40a5-a0d6-ab42d31d8ea2-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:03.879491 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.879456    2567 generic.go:356] "Generic (PLEG): container finished" podID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerID="858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837" exitCode=0
Jul 06 16:18:03.879955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.879506    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"
Jul 06 16:18:03.879955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.879534    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerDied","Data":"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"}
Jul 06 16:18:03.879955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.879578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n" event={"ID":"8572acad-677b-40a5-a0d6-ab42d31d8ea2","Type":"ContainerDied","Data":"28422459b00b264803080e7de3be37d8b7e2d02e0a29abf26b8514011d68f503"}
Jul 06 16:18:03.879955 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.879601    2567 scope.go:117] "RemoveContainer" containerID="858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"
Jul 06 16:18:03.888431 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.888405    2567 scope.go:117] "RemoveContainer" containerID="8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"
Jul 06 16:18:03.896120 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.896103    2567 scope.go:117] "RemoveContainer" containerID="858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"
Jul 06 16:18:03.896332 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:18:03.896315    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837\": container with ID starting with 858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837 not found: ID does not exist" containerID="858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"
Jul 06 16:18:03.896383 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.896341    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837"} err="failed to get container status \"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837\": rpc error: code = NotFound desc = could not find container \"858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837\": container with ID starting with 858e0a6e63d5991253e41f3f1adda3b3a1206f627b272d283699b424dd19d837 not found: ID does not exist"
Jul 06 16:18:03.896383 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.896361    2567 scope.go:117] "RemoveContainer" containerID="8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"
Jul 06 16:18:03.896606 ip-10-0-138-21 kubenswrapper[2567]: E0706 16:18:03.896592    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5\": container with ID starting with 8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5 not found: ID does not exist" containerID="8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"
Jul 06 16:18:03.896649 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.896610    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5"} err="failed to get container status \"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5\": rpc error: code = NotFound desc = could not find container \"8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5\": container with ID starting with 8041f59f21fce78e35e0f5f01fdfc422c3ce92d8f68ed7847aac6b5f4cbb2cf5 not found: ID does not exist"
Jul 06 16:18:03.907241 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.907216    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:18:03.912915 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:03.912896    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c8a9d1-openshift-default-7f99949595-4x94n"]
Jul 06 16:18:05.664635 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:05.664596    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" path="/var/lib/kubelet/pods/8572acad-677b-40a5-a0d6-ab42d31d8ea2/volumes"
Jul 06 16:18:06.786568 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:06.786534    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:18:06.787031 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:06.786780    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" containerID="cri-o://56849435299b68df94e8adaa9f06d1cb9fcc03e7ea77f3266287ad69317970ca" gracePeriod=30
Jul 06 16:18:10.328784 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:10.328741    2567 patch_prober.go:28] interesting pod/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 16:18:10.329187 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:10.328836    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 16:18:11.921212 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:11.921182    2567 generic.go:356] "Generic (PLEG): container finished" podID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerID="56849435299b68df94e8adaa9f06d1cb9fcc03e7ea77f3266287ad69317970ca" exitCode=0
Jul 06 16:18:11.921571 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:11.921259    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerDied","Data":"56849435299b68df94e8adaa9f06d1cb9fcc03e7ea77f3266287ad69317970ca"}
Jul 06 16:18:11.921571 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:11.921326    2567 scope.go:117] "RemoveContainer" containerID="10b6aafb8f3584fba46bf56cc0bc83c045ba9225614a4efd28420eae20773c3d"
Jul 06 16:18:12.043285 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.043256    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:18:12.172585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172496    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172548    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5jdh\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172585 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172834 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172753    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172834 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172786    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172834 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172811    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172998 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172843    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172998 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172866    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:12.172998 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.172998 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.172920    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs\") pod \"4a71d6b7-833c-43fc-9a62-336a0935659d\" (UID: \"4a71d6b7-833c-43fc-9a62-336a0935659d\") "
Jul 06 16:18:12.173206 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.173171    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 16:18:12.173381 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.173333    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:12.173490 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.173386    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:12.173647 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.173596    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.173647 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.173626    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a71d6b7-833c-43fc-9a62-336a0935659d-istiod-ca-cert\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.174078 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.174052    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:12.175124 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.175101    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh" (OuterVolumeSpecName: "kube-api-access-v5jdh") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "kube-api-access-v5jdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:18:12.175124 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.175117    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:18:12.175266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.175201    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 16:18:12.175266 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.175203    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a71d6b7-833c-43fc-9a62-336a0935659d" (UID: "4a71d6b7-833c-43fc-9a62-336a0935659d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 16:18:12.274937 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274900    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-workload-certs\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.274937 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274930    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5jdh\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-kube-api-access-v5jdh\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.274937 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274940    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-token\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.275180 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274950    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-podinfo\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.275180 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274958    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-envoy\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.275180 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274965    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-istio-data\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.275180 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.274972    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a71d6b7-833c-43fc-9a62-336a0935659d-credential-socket\") on node \"ip-10-0-138-21.ec2.internal\" DevicePath \"\""
Jul 06 16:18:12.323330 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.323293    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gk459_af73161c-e2d8-4380-966b-7f04591726e6/manager/0.log"
Jul 06 16:18:12.452439 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.452412    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9lllh_e57cc142-4bec-42eb-8a44-511b68d2b09e/manager/0.log"
Jul 06 16:18:12.576001 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.575970    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5d86f67b86-82gjj_d1eab4a6-e317-4f6c-8541-c1114a01b86d/maas-api/0.log"
Jul 06 16:18:12.927798 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.927717    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"
Jul 06 16:18:12.928249 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.927715    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l" event={"ID":"4a71d6b7-833c-43fc-9a62-336a0935659d","Type":"ContainerDied","Data":"0cb349acbc82d7146d1be69008c8c0d09ac47d18755afca4a809370d0d6bdb2a"}
Jul 06 16:18:12.928249 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.927857    2567 scope.go:117] "RemoveContainer" containerID="56849435299b68df94e8adaa9f06d1cb9fcc03e7ea77f3266287ad69317970ca"
Jul 06 16:18:12.929971 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.929952    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94f47f95b-4wh4b_3a142576-4499-48f4-9984-6c1f3b29a62e/manager/0.log"
Jul 06 16:18:12.951831 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.951804    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:18:12.957062 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:12.957035    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b747bf-openshift-default-c57bf9b55-7h85l"]
Jul 06 16:18:13.183964 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:13.183878    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8g5d5_1d8c4500-9880-4dfc-b9bb-9a80833ece28/manager/0.log"
Jul 06 16:18:13.308010 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:13.307979    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-47q8t_9199c637-72b2-405f-a477-481ace41076e/copy-manifests/0.log"
Jul 06 16:18:13.316750 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:13.316706    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-47q8t_9199c637-72b2-405f-a477-481ace41076e/manager/0.log"
Jul 06 16:18:13.669887 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:13.669857    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" path="/var/lib/kubelet/pods/4a71d6b7-833c-43fc-9a62-336a0935659d/volumes"
Jul 06 16:18:13.723046 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:13.723016    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-twzdq_93467ae9-a059-41cc-a755-10d176a0d407/postgres/0.log"
Jul 06 16:18:15.205124 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:15.205089    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gbhqq_0b0a5ad8-6685-442a-993b-0af67a573139/manager/0.log"
Jul 06 16:18:15.430065 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:15.430029    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-llrkc_6587014d-7185-484b-a6e8-15a29fb2b75f/kuadrant-console-plugin/0.log"
Jul 06 16:18:15.545052 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:15.544976    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8cb9d_2642697c-4919-4fb3-9b50-69dbdf2d40d9/registry-server/0.log"
Jul 06 16:18:15.888855 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:15.888767    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5wnfn_7d67df61-0e35-43f7-9edc-93bcbc9c49f5/manager/0.log"
Jul 06 16:18:16.244204 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:16.244179    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tcldv_4b47c5e9-014c-45a1-a6aa-506dbf588bd9/istio-proxy/0.log"
Jul 06 16:18:16.581193 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:16.581107    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745c454698-g7b2h_40161411-ffd3-4958-b82c-569fa110d209/kube-auth-proxy/0.log"
Jul 06 16:18:16.694925 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:16.694872    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/1.log"
Jul 06 16:18:16.809127 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:16.809092    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpbqn_d31f62bf-ab10-497c-8e2b-d3c2bdc99387/payload-pre-processing/0.log"
Jul 06 16:18:16.947177 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:16.947145    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-f8pfl_6766022a-d3f6-46af-b15d-9c854546db52/payload-processing/0.log"
Jul 06 16:18:17.454679 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:17.454643    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6jxfx_27b51d3e-371a-469b-b9b5-f763f6cd5177/main/0.log"
Jul 06 16:18:17.794537 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:17.794468    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6w84b_66d2b339-fd7e-4637-b558-31c84779d276/main/0.log"
Jul 06 16:18:18.032634 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:18.032602    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-pkrxj_8bf23871-faa3-423f-899e-5d79b0b437b4/main/0.log"
Jul 06 16:18:26.055428 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:26.055393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d48l4_eb24f93f-b357-4b8d-a604-6d89306a5dd6/global-pull-secret-syncer/0.log"
Jul 06 16:18:26.230749 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:26.230717    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ffjm5_92943c63-f326-47cb-819d-16d5f00081e3/konnectivity-agent/0.log"
Jul 06 16:18:26.364396 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:26.364314    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-21.ec2.internal_57895fd63ca76ed1e35265f2fb017564/haproxy/0.log"
Jul 06 16:18:31.650879 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:31.650838    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gbhqq_0b0a5ad8-6685-442a-993b-0af67a573139/manager/0.log"
Jul 06 16:18:31.702830 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:31.702800    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-llrkc_6587014d-7185-484b-a6e8-15a29fb2b75f/kuadrant-console-plugin/0.log"
Jul 06 16:18:31.747328 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:31.747278    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8cb9d_2642697c-4919-4fb3-9b50-69dbdf2d40d9/registry-server/0.log"
Jul 06 16:18:31.975058 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:31.975019    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5wnfn_7d67df61-0e35-43f7-9edc-93bcbc9c49f5/manager/0.log"
Jul 06 16:18:34.119398 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.119368    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-state-metrics/0.log"
Jul 06 16:18:34.157337 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.157311    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-rbac-proxy-main/0.log"
Jul 06 16:18:34.181383 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.181346    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dgn8m_87e1bded-e130-47fd-acde-9860ce5d753d/kube-rbac-proxy-self/0.log"
Jul 06 16:18:34.242302 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.242276    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-mb8wc_16183bad-7781-4738-a10e-989bcc171f4e/monitoring-plugin/0.log"
Jul 06 16:18:34.375651 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.375567    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/node-exporter/0.log"
Jul 06 16:18:34.428883 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.428856    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/kube-rbac-proxy/0.log"
Jul 06 16:18:34.478968 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:34.478942    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cvf4k_9c0430c8-7e38-49ba-84b4-955d2b31b9d8/init-textfile/0.log"
Jul 06 16:18:36.421627 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.421597    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"]
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.421968    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.421979    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.421990    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b63c451-c60f-4308-9650-a59d5c0dc023" containerName="main"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.421996    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b63c451-c60f-4308-9650-a59d5c0dc023" containerName="main"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422004    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422009    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422019    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422019 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422024    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422311 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422082    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422311 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422089    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a71d6b7-833c-43fc-9a62-336a0935659d" containerName="istio-proxy"
Jul 06 16:18:36.422311 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422095    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b63c451-c60f-4308-9650-a59d5c0dc023" containerName="main"
Jul 06 16:18:36.422311 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422101    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy"
Jul 06 16:18:36.422311 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.422107    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8572acad-677b-40a5-a0d6-ab42d31d8ea2" containerName="istio-proxy"
Jul 06 16:18:36.425327 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.425307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.430616 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.430581    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z6r42\"/\"default-dockercfg-mwdkp\""
Jul 06 16:18:36.430750 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.430619    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6r42\"/\"openshift-service-ca.crt\""
Jul 06 16:18:36.431875 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.431732    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6r42\"/\"kube-root-ca.crt\""
Jul 06 16:18:36.446146 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.446122    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"]
Jul 06 16:18:36.494087 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.494058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjvj\" (UniqueName: \"kubernetes.io/projected/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-kube-api-access-8kjvj\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.494288 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.494104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-podres\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.494288 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.494128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-proc\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.494288 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.494216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-sys\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.494431 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.494310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-lib-modules\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595679 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595624    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-podres\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595842 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-proc\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595842 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-sys\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595842 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595762    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-proc\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595842 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595790    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-podres\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.595842 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-lib-modules\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.596052 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjvj\" (UniqueName: \"kubernetes.io/projected/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-kube-api-access-8kjvj\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.596052 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595864    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-sys\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.596052 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.595943    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-lib-modules\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.606455 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.606429    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjvj\" (UniqueName: \"kubernetes.io/projected/71c2daef-652f-4fe9-8a13-0c1fa6e26c3b-kube-api-access-8kjvj\") pod \"perf-node-gather-daemonset-gd7m5\" (UID: \"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.735538 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.735504    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:36.893911 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:36.893886    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"]
Jul 06 16:18:36.896685 ip-10-0-138-21 kubenswrapper[2567]: W0706 16:18:36.896625    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod71c2daef_652f_4fe9_8a13_0c1fa6e26c3b.slice/crio-a9cf7569dd6bae6e828562c6bedd4aa85598f457252a2603bfe7ed65e8c01bfa WatchSource:0}: Error finding container a9cf7569dd6bae6e828562c6bedd4aa85598f457252a2603bfe7ed65e8c01bfa: Status 404 returned error can't find the container with id a9cf7569dd6bae6e828562c6bedd4aa85598f457252a2603bfe7ed65e8c01bfa
Jul 06 16:18:37.040291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:37.040197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5" event={"ID":"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b","Type":"ContainerStarted","Data":"94f793c7a3c4e137653e5691846cd0f5bcf8e4ab6e6f03695acec3c500b0f9e4"}
Jul 06 16:18:37.040291 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:37.040252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5" event={"ID":"71c2daef-652f-4fe9-8a13-0c1fa6e26c3b","Type":"ContainerStarted","Data":"a9cf7569dd6bae6e828562c6bedd4aa85598f457252a2603bfe7ed65e8c01bfa"}
Jul 06 16:18:37.040500 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:37.040342    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:37.063044 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:37.062997    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5" podStartSLOduration=1.06298096 podStartE2EDuration="1.06298096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:18:37.061686864 +0000 UTC m=+2683.931734025" watchObservedRunningTime="2026-07-06 16:18:37.06298096 +0000 UTC m=+2683.933028184"
Jul 06 16:18:40.160720 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:40.160691    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8gbx2_1420a3f1-5a18-435f-bc21-6fe18f4f151a/dns/0.log"
Jul 06 16:18:40.184438 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:40.184409    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8gbx2_1420a3f1-5a18-435f-bc21-6fe18f4f151a/kube-rbac-proxy/0.log"
Jul 06 16:18:40.272692 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:40.272641    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5ts55_2cb8845b-4fb5-40c1-b7ae-399f78bf1d94/dns-node-resolver/0.log"
Jul 06 16:18:40.923614 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:40.923575    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-df74c4f86-5mjbl_7a995b57-0715-428d-ad2e-05a005ec8aae/registry/0.log"
Jul 06 16:18:40.991142 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:40.991115    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7gdp_c60ca798-3c9a-48dd-8565-fd717dadb1e1/node-ca/0.log"
Jul 06 16:18:42.064365 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.064337    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tcldv_4b47c5e9-014c-45a1-a6aa-506dbf588bd9/istio-proxy/0.log"
Jul 06 16:18:42.243088 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.243064    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745c454698-g7b2h_40161411-ffd3-4958-b82c-569fa110d209/kube-auth-proxy/0.log"
Jul 06 16:18:42.277691 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.277647    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/0.log"
Jul 06 16:18:42.303364 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.303333    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/1.log"
Jul 06 16:18:42.334645 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.334568    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpbqn_d31f62bf-ab10-497c-8e2b-d3c2bdc99387/payload-pre-processing/0.log"
Jul 06 16:18:42.383938 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:42.383912    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-f8pfl_6766022a-d3f6-46af-b15d-9c854546db52/payload-processing/0.log"
Jul 06 16:18:43.055329 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:43.055289    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-gd7m5"
Jul 06 16:18:43.156466 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:43.156433    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-284nb_0b0d376a-ab7d-4d85-a086-ec9220f2b072/serve-healthcheck-canary/0.log"
Jul 06 16:18:43.788195 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:43.788123    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2d54_25e27276-9bc9-4f81-85ae-5c2fce17e039/kube-rbac-proxy/0.log"
Jul 06 16:18:43.817014 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:43.816984    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2d54_25e27276-9bc9-4f81-85ae-5c2fce17e039/exporter/0.log"
Jul 06 16:18:43.849587 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:43.849561    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2d54_25e27276-9bc9-4f81-85ae-5c2fce17e039/extractor/0.log"
Jul 06 16:18:46.847422 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:46.847340    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gk459_af73161c-e2d8-4380-966b-7f04591726e6/manager/0.log"
Jul 06 16:18:46.920516 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:46.920483    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tcldv_4b47c5e9-014c-45a1-a6aa-506dbf588bd9/istio-proxy/0.log"
Jul 06 16:18:46.982428 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:46.982397    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9lllh_e57cc142-4bec-42eb-8a44-511b68d2b09e/manager/0.log"
Jul 06 16:18:47.026446 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.026414    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5d86f67b86-82gjj_d1eab4a6-e317-4f6c-8541-c1114a01b86d/maas-api/0.log"
Jul 06 16:18:47.081877 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.081853    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745c454698-g7b2h_40161411-ffd3-4958-b82c-569fa110d209/kube-auth-proxy/0.log"
Jul 06 16:18:47.103516 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.103428    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/0.log"
Jul 06 16:18:47.130418 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.130393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/1.log"
Jul 06 16:18:47.163207 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.163174    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpbqn_d31f62bf-ab10-497c-8e2b-d3c2bdc99387/payload-pre-processing/0.log"
Jul 06 16:18:47.213937 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.213901    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-94f47f95b-4wh4b_3a142576-4499-48f4-9984-6c1f3b29a62e/manager/0.log"
Jul 06 16:18:47.229299 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.229267    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-f8pfl_6766022a-d3f6-46af-b15d-9c854546db52/payload-processing/0.log"
Jul 06 16:18:47.315080 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.315044    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8g5d5_1d8c4500-9880-4dfc-b9bb-9a80833ece28/manager/0.log"
Jul 06 16:18:47.426223 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.426144    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-47q8t_9199c637-72b2-405f-a477-481ace41076e/manager/0.log"
Jul 06 16:18:47.453411 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.453388    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-47q8t_9199c637-72b2-405f-a477-481ace41076e/copy-manifests/0.log"
Jul 06 16:18:47.697721 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:47.697701    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-twzdq_93467ae9-a059-41cc-a755-10d176a0d407/postgres/0.log"
Jul 06 16:18:48.594280 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.594246    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tcldv_4b47c5e9-014c-45a1-a6aa-506dbf588bd9/istio-proxy/0.log"
Jul 06 16:18:48.735039 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.734949    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-745c454698-g7b2h_40161411-ffd3-4958-b82c-569fa110d209/kube-auth-proxy/0.log"
Jul 06 16:18:48.761747 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.761705    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/0.log"
Jul 06 16:18:48.787699 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.787658    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5c6vz_fea629bd-0b5c-44bf-bdfb-5ad5d3cbe3a0/istio-proxy/1.log"
Jul 06 16:18:48.813601 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.813573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpbqn_d31f62bf-ab10-497c-8e2b-d3c2bdc99387/payload-pre-processing/0.log"
Jul 06 16:18:48.860095 ip-10-0-138-21 kubenswrapper[2567]: I0706 16:18:48.860065    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-f8pfl_6766022a-d3f6-46af-b15d-9c854546db52/payload-processing/0.log"